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Executive  Summary 


Historically,  live-fire  testing  (LFT)  was  conceived  to  detennine  survivability  and  lethality  of  an 
autonomous  combat  platform  in  a  combined  arms  team  operating  in  a  Cold  War  setting.  In  this 
context,  LFT  has  been  invaluable  for  identifying  strengths  and  weaknesses  in  system  and 
munition  design,  specifying  limits  of  platform  combat  survivability,  and  validating  analytical 
models. 

Despite  this  success,  the  variety  of  threats  and  combat  landscapes  facing  today’s  warfighter  and 
the  increased  sophistication  of  platforms  operating  within  a  system  of  systems  (SoS)  context  in 
an  integrated  and  information-centric  battlefield  suggests  the  need  for  a  new  look  at  how  live-fire 
test  and  evaluation  (LFT&E)  programs  are  conducted.  In  particular: 

•  How  should  LFT&E  planning  and  execution  change  in  light  of  SoS  tactical  doctrines? 

•  How  should  LFT&E  be  designed  and  conducted  to  (1)  collect  data  relevant  to 
vulnerability/lethality  (V/L)  assessment  decisions,  (2)  generate  optimal  combinations  of 
component-,  subsystem-,  and  system-level  test  data,  considering  cost,  availability  of 
hardware,  and  production  schedule,  and  (3)  evaluate  LFT  results  so  decision-makers  can 
ascertain  mission  success  in  the  joint  environment? 

LFT&E  has  focused  on  critical  issues  associated  with  the  tested  platform,  typically  of  the  form: 

•  What  is  the  vulnerability  of  the  platfonn  against  the  spectrum  of  current  and  future  threats 
as  identified  by  the  intelligence  community? 

•  What  vulnerability  reduction  measures  increase  crew,  passenger,  and  system  survivability? 

•  How  effective  is  battle  damage  assessment  and  repair  (BDAR)  in  restoring  the  platform  to 
functional  combat  capability? 

In  the  context  of  the  V/L  taxonomy  these  issues  have  been  addressed  almost  exclusively  at 
Levels  2  (i.e.,  component  status)  and  4  (i.e.,  mission  utility).  Only  to  a  limited  extent  have 
results  been  related  to  Level  3  (i.e.,  capability  status).  This  information,  cast  in  terms  of 
probabilities  of  mission  or  catastrophic  kills,  allows  only  a  generalization  of  the  results  in  terms 
of  a  global  ability  to  complete  the  spectrum  of  missions  likely  to  be  assigned  to  the  platform. 

But,  while  remaining  mission  utility  has  been  represented  in  the  context  of  a  unit  assigned  to 
prosecute  a  generic  mission  intended  for  the  tested  platform,  the  platform  has  been  considered 
autonomous.  In  that  sense,  concern  has  focused  on  whether  the  damaged  platform  can  continue 
an  assigned  mission  or  perhaps  through  BDAR  be  used  for  an  alternate  mission.  Damage  has 
seldom  been  evaluated  in  the  context  of  specific  missions  or  complementing  capabilities  of  other 
platforms  within  the  mission-prosecuting  unit. 
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Within  the  SoS  environment,  the  objective  of  test  and  evaluation  (T&E)  is  to  facilitate  the 
measurement  and  assessment  of  (1)  the  effectiveness,  suitability,  and  survivability  of  platforms 
relevant  to  their  contributions  to  the  SoS  and  (2)  the  effectiveness  of  the  SoS  in  achieving 
mission  objectives  at  the  Joint  Forces  Command  (JFC)  level.  T&E,  therefore,  requires  the 
assessment  of  the  capabilities  of  SoS  platforms,  individually  and  collectively,  to  complete 
identified  tasks  in  realistic  scenarios. 

The  Missions  and  Means  Framework  (MMF)  provides  a  structure  for  linking  mission  tasks  to  the 
capabilities  required  for  task  completion  and  supplies  a  disciplined  procedure  for  the 
identification  of  the  means  to  achieve  mission  tasks  and  the  assessment  of  mission 
accomplishment.  An  MMF-based  SoS  task-focused  FFT&E  strategy  is  proposed  to  replace  the 
traditional  platform-centric  strategy  that  emphasizes  the  functional  capabilities  of  the 
autonomous  platform.  The  MMF-based  strategy  focuses  on  the  extent  to  which  the  platform 
retains  those  capabilities  needed  for  completion  of  SoS  tasks  and  the  ability  of  the  SoS  to 
complete  current  and  future  mission  tasks  given  the  residual  and  available  capabilities  of  the 
platform  following  ballistic  damage. 

The  table  illustrates  three  major  areas  in  which  the  traditional  platform-centric  FFT&E  strategy 
is  expected  to  differ  from  the  proposed  MMF-based  SoS  task-focused  FFT&E  strategy. 

The  traditional  platform-centric  strategy  focuses  on  the  extent  to  which  a  platform  retains 
battlefield  combat  utility  (Fevel  4)  or  a  general  ability  to  complete  missions  likely  to  be  assigned 
to  the  platform  when  subjected  to  current  and  expected  future  threats.  The  platform  is 
considered  autonomous,  and  little  consideration  is  given  in  the  critical  issues  to  the 
complementary  capabilities  of  other  platforms  that  are  part  of  the  SoS  or  to  the  platform’s  role  in 
the  completion  of  tasks  that  are  linked  to  specific  missions  in  the  joint  environment. 

In  contrast,  the  MMF-based  SoS  task-focused  strategy  examines  the  capabilities  of  the  platform 
within  the  context  of  the  SoS  (Fevel  3)  and  the  extent  to  which  damage  to  the  platform  from 
current  and  expected  future  threats  affects  the  SoS’s  ability  (Fevel  3)  to  complete  specific 
mission  tasks  (Fevel  4).  Redundancies  and  interdependencies  among  SoS  platforms  are 
considered  in  the  identification  of  critical  issues  and  prioritization  of  data  voids. 

With  limited  time  and  dollars  to  devote  to  EFT,  shotline  selection  requires  the  prioritization  of 
identified  data  voids  —  areas  in  which  there  is  little  understanding  of  the  effects  of  ballistic 
interactions  on  platform  capabilities.  In  the  MMF-based  SoS  task-focused  strategy,  the  limited 
EFT  shots  must  be  “spent”  to  address  the  most  urgent  questions  of  a  platfonn’s  ability  to  support 
the  SoS  mission  tasks.  The  technical  risk  associated  with  failing  to  address  less  critical 
capabilities  in  an  EFT  may  be  examined  via  modeling  and  simulation. 

In  the  MMF-based  SoS  task-focused  strategy,  however,  the  focus  is  the  capabilities  of  the 
platform  within  the  context  of  the  SoS  and  the  ability  to  complete  SoS  mission  tasks.  The 
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Table.  Comparison  of  traditional  and  MMF-based  LFT&E  strategies. 


Traditional  Platform-Centric  LFT&E 
Strategy 

MMF-Based  SoS  Task-Focused  LFT&E 
Strategy 

Critical  Issues 

What  is  the  vulnerability  of  the  platform  to 

current  and  future  threats  identified  by  the 
intelligence  community? 

What  is  the  reduction  in  the  ability  of  the  SoS 
to  prosecute  typical  missions  after  damage 
from  current  and  future  threats  identified  by  the 
intelligence  community? 

Flow  effective  is  BDAR  in  restoring  the 

platform  to  functional  capability  after  an 
attack? 

Flow  effective  are  BDAR  and  other 
maintenance  actions  in  restoring  SoS 
capabilities  critical  to  mission  prosecution 
after  an  attack? 

Shotline 

Selection 

Platform:  Based  on  technical  risk  associated 
with  the  inability  to  determine  platform 
capability  as  the  result  of  ballistic  damage. 

Platform:  Based  on  technical  risk  associated 
with  the  inability  to  determine  the  effect  on 
mission  prosecution  caused  by  loss  of 
platform  capabilities  as  the  result  of  ballistic 

damage. 

Damage 

Assessment 

Platform: 

•  Map  to  remaining  combat  utility  via 
Damage  Assessment  List  (DAL)  or  other 
03,4  construct. 

Platform: 

•  Map  to  remaining  SoS  capabilities  by 
analysis  and  operational-type  tests. 

Crew: 

•  Map  crew  incapacitation  to  remaining 
combat  utility  via  DAL  or  other  03,4 
construct. 

Crew: 

•  Map  crew  incapacitation  to  platform  loss  of 
capabilities  and  confirm  remaining  SoS 
capabilities  by  analysis  and  operational- 
type  tests. 

BDAR:  Determine  expedient  repairs  that  can  be 
made  to  restore  platform  to  some  level  of 
combat  utility. 

Mission  Damage  Assessment  and  Repair: 

•  BDAR:  Determine  expedient  repairs  that  can 
be  made  to  restore  some  platform 
capabilities  during  and  immediately 
following  an  engagement. 

•  Other  maintenance  procedures:  Conduct 
further  repair  to  anticipate  future 
engagements  during  the  mission. 

remaining  capabilities  following  an  LFT  can  be  determined  in  laboratory  settings  and  through 
quasi-operational  tests  using  companion  vehicles  to  confirm  the  usefulness  of  residual 
capabilities  within  the  SoS  construct  or  to  develop  and  validate  “workarounds”  to  complete 
mission  tasks  with  capabilities  remaining  within  the  SoS  (i.e.,  damaged  platform  plus  companion 
platforms). 

Thus,  the  emphasis  of  the  proposed  MMF-based  strategy  on  SoS  tasks  is  reflected  in  both  the 
planning  for  LFT  and  the  evaluation  of  LFT  results.  In  LFT  planning,  critical  issues  that 
emphasize  recoverability  are  identified  with  the  perspective  directed  toward  SoS  operations  in 
the  joint  environment.  Modifications  to  shotline  selection  and  damage  assessment  are  expected, 
as  the  focus  expands  to  include  the  long-term,  as  well  as  the  short-term,  needs  for  SoS 
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capabilities.  The  strategy  allows  for  a  realistic  assessment  of  the  technical  risk  associated  with 
foregoing  test  shots  that  may  be  of  interest  at  the  platform  level  but  are  not  critical  to 
understanding  SoS  effectiveness.  In  evaluation,  the  roll-up  of  platform  LFT  program  results  to 
the  SoS  level  provides  decision-makers  with  a  better  grasp  of  the  ability  of  the  unit  of  operation 
to  complete  tasks  to  standards  under  given  conditions  and  the  risks  associated  with  alternative 
courses  of  action. 

With  consideration  to  the  risks  associated  with  vulnerability  assessment,  a  structured  process  for 
building  cost-effective  LFT&E  programs  in  an  MMF  environment  is  presented.  This 
methodology  includes  the  identification  and  prioritization  of  data  voids  and  the  selection  of  the 
optimal  program  elements  for  addressing  those  voids,  considering  time,  production  schedule, 
hardware  availability,  and  cost.  Activity-based  costing  (ABC),  proposed  as  the  appropriate 
methodology  for  costing  LFT&E  program  elements,  allows  the  value  added  in  completing 
LFT&E  program  elements,  activities,  and  sub-activities  to  be  weighed  against  the  costs  incurred. 
The  identification,  measurement,  and  reporting  of  costs  according  to  established  standards, 
consistent  across  time  periods  and  comparable  across  systems,  and  the  establishment  of  a 
database  accessible  for  purposes  of  estimating  costs  pre-test  and  evaluating  cost  variances  post¬ 
test  are  recommended. 

Designing  a  cost-effective  LFT&E  program  requires  meeting  LFT&E  objectives  with 
consideration  given  to  the  limited  resources  available.  ABC  provides  the  framework  for  the 
decision-maker  to  view  LFT&E  program  elements  from  both  strategic  and  operational 
perspectives,  eliminating  non-value-added  elements  and  activities  and  seeking  ways  to  make 
value-added  elements  and  activities  more  efficient  and  effective. 
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1.  Introduction 


1.1  Background 

Live-fire  testing  (LFT)  is  now  in  its  third  decade.  Motivated  by  Joint  Live-Fire  (JLF)  testing  of 
front-line  U.S.  platforms  and  munitions  beginning  in  1984,  Live-Fire  Test  and  Evaluation  (LFT&E) 
became  a  fonnal  part  of  the  Department  of  Defense  (DoD)  acquisition  process  in  1987. 

U.S.  Code  (Title  10  U.S.  Code  Section  2366)  mandates  LFT&E  for  covered  systems,  major 
munitions  programs,  missile  programs,  and  product  improvements  to  covered  systems  before  they 
can  proceed  beyond  low-rate  initial  production  (LRIP).  The  objective  of  an  LFT&E  program  is  to 
support  a  “timely  and  thorough  assessment  of  the  vulnerability/lethality  of  a  system  as  it  progresses 
through  its  development  and  subsequent  production  phases.”  An  effective  LFT&E  program 
demonstrates  “the  ability  of  the  weapon  system  or  munition  to  provide  battle  resilient  survivability^ 
or  lethality  and  provide[s]  insights  into  [both]  the  principal  damage  mechanisms  and  failure  modes 
occurring  as  a  result  of  the  munition/target  interaction  and  . . .  [the]  techniques  for  reducing  personnel 
casualties  or  enhancing  system  survivability/lethality  (DA  Pam  73-1,  p.  198).” 

A  vulnerability  or  lethality  assessment  strategy*  established  early  in  the  development  of  a  weapon 
system  has  the  potential  to  detect  system  vulnerabilities  or  deficits  in  system’s  lethality,  as  well  as  to 
reduce  or  minimize  the  costs  associated  with  combat  losses  and  retrofits  of  systems  in  the  late  stages 
of  production.  An  LFT&E  program  is  designed  with  the  purpose  of  collecting  data  relative  to  certain 
vulnerability/lethality  (V/L)  characteristics  of  the  platfonn  or  munition.  A  typical  program  includes 
various  combinations  of  coupon-,  component-,  subsystem-,  and  system-level  tests  and  is  supported 
by  data  from  modeling  and  simulation  (M&S);  design  analyses;  analyses  of  combat,  safety,  and 
mishaps;  controlled  damage  experiments  (CDE);  and  developmental  and  operational  tests.8  In  this 
report,  it  is  assumed  that  the  focus  of  LFT&E  programs  is  ballistic  threat  effects.** 


* 

A  covered  system  is  an  Acquisition  Category  I  or  II  program  vehicle,  weapon  platform,  or  conventional  weapon  system  that 
includes  features  designed  to  provide  some  degree  of  protection  to  users  in  combat.  A  commercial  or  nondevelopmental  item 
may  be  a  covered  system  or  a  part  of  a  covered  system,  depending  upon  its  intended  use. 

^Survivability  is  “the  capability  of  a  system  and  crew  to  avoid  or  withstand  a  manmade  hostile  environment  without  suffering 
an  abortive  impairment  of  its  ability  to  accomplish  the  designated  mission.”  Survivability  consists  of  susceptibility,  vulnerability, 
and  recoverability.  The  focus  of  the  LFT  program  is  vulnerability  (i.e.,  kill  given  a  hit)  (DA  Pam  73-1,  p.  296  [2003]). 

fin  this  report,  vulnerability  (lethality)  assessment  strategy  is  defined  as  the  overall  plan  designed  with  the  purpose  of 
collecting  sufficient  relevant  data  for  evaluators  to  assess  a  system’s  vulnerability  (lethality).  As  the  term  in  used  in  this  report, 
the  strategy  includes  activities  conducted  by  government  and  nongovernment  analysts  and  testers,  as  well  as  government  evaluators. 

8 A  developmental  test  is  a  “generic  term  encompassing  M&S  and  engineering  type  tests  that  are  used  to  verify  that  design 
risks  are  minimized,  that  safety  of  the  system  is  certified,  that  achievement  of  system  technical  performance  is  substantiated,  and 
that  readiness  for  operational  T&E  is  certified.”  LFT&E  may  be  classified  as  a  developmental  test  (Army  Regulation  73-1, 
p.  17  [2004]). 

An  operational  test  is  a  “field  test  of  a  system. .  .under  realistic  operational  conditions  with  users  who  represent  those  expected 
to  operate  and  maintain  the  system  when  it  is  fielded  or  deployed”  (Army  Regulation  73-1,  p.  21  [2004]). 

It  is  recognized  that  V/L  assessment  strategies  must  consider  the  effects  from  ballistic  and  nonballistic  threats.  Some 
LFT&E  programs  are  extended  to  include  certain  conventional  nonballistic  threats,  such  as  lasers  and  high-powered  microwaves. 
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As  part  of  the  V/L  assessment  strategy  for  a  system  included  under  the  U.S.  congressional 
LFT&E  mandate,  a  Full-Up  System-Level  (FUSL)  LFT&E  must  be  completed  by  independent 
agencies  prior  to  the  system  entering  full-scale  production.’'  A  FUSL  LFT  includes  “realistic 
survivability  testing  . .  .testing  for  vulnerability  of  the  system  in  combat  by  firing  munitions  likely 
to  be  encountered  in  combat... at  the  system  configured  for  combat”  or  “realistic  lethality 
testing. .  .testing  for  lethality  by  firing  the  munition  or  missile  concerned  at  appropriate  targets 
configured  for  combat  (Title  10  U.S.  Code  Section  2366).” 

Because  significant  resources  may  be  consumed  in  its  planning,  execution,  and  evaluation,  it  is 
important  to  understand  the  role  of  FUSL  LFT  in  a  V/L  assessment  strategy  and  weigh  the  costs 
and  benefits  of  competing  alternative  assessment  plans./  The  Secretary  of  Defense  may  waive 
the  application  of  tests,  if  the  Secretary  certifies  to  Congress  that  a  FUSL  LFT  of  such  a  system 
or  program  would  be  unreasonably  expensive  and  impractical.  A  waiver  and  alternative  LFT&E 
strategy  must  be  submitted  and  approved  by  Milestone  B  (i.e.,  approval  to  enter  System 
Development  &  Demonstration  Phase)  (see  figure  1). 


Figure  1.  Defense  acquisition  time  line. 


*  .  .  . 

A  LFT  is  defined  as  a  test  within  the  LFT&E  program  approved  by  the  Office  of  the  Secretary  of  Defense  and  includes  firing 
of  actual  munitions  at  target  components,  subsystems,  or  system-level  targets  to  examine  personnel  casualty,  vulnerability,  or 
lethality  issues.  A  FUSL  LFT  is  defined  as  the  “testing  that  fully  satisfies  the  statutory  requirement  for  realistic”  (in  threat, 
configuration  of  target,  and  operating  environment)  “survivability  testing  or  realistic  lethality  testing  (as  defined  in  Title  10  of  the 
USC)  (DA  Pam  73-1,  p.  203).” 

fin  LFT  legislation,  an  alternative  strategy  is  a  strategy  that  does  not  include  FUSL  LFT.  In  this  report,  however,  the  term 
alternative,  as  in  alternative  strategy  or  alternative  plan,  is  defined  more  broadly  and  is  used  to  indicate  more  than  one  option  or 
a  choice  among  options.  Therefore,  one  or  more  alternative  V/L  assessment  strategies  or  plans  may  include  FUSL  LFT  as  an 
element. 
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1.2  Contributions  of  LFT&E  Programs 

Since  1985,  more  than  35  U.S.  Army  platforms  and  munitions  have  undergone  FUSL  testing 
under  the  JLF  or  Congressional-mandated  LFT  programs.*  The  focus  over  this  period  has  been 
on  the  V/L  aspects  of  platform  or  munition  perfonnance  in  a  combined  arms  team  in  a  Cold  War 
setting.  During  this  period,  improvements  in  test  planning  and  execution  have  led  to  an  increase 
in  the  credibility  of  test  results.  Criticisms  leveled  against  the  Services  during  Bradley  LFT  in 
1985-86  motivated  examinations  of  how  system-level  vulnerability  and  lethality  tests  were 
planned  and  conducted.  Plagued  by  a  plethora  of  threat  munitions  (in  vulnerability  tests), 
platform  design  weaknesses  potentially  contributing  to  crew  casualties  and  system  vulnerability, 
and  combat  conditions  to  be  replicated,  test  planners  developed  systematic  processes  for  shotline 
selection,  shot  ordering,  and  damage  assessment  that  were  derived  from  pre-test  identification  of 
critical  issues  for  the  LFT. 

Identification  of  the  critical  issues  is  the  key  to  LFT  success.  Only  with  a  set  of  critical  issues 
can  the  tester  hope  to  extract  test  data  that  are  relevant,  useful,  and  necessary.  Test  execution 
requires  discipline  to  collect  data  of  the  resolution  and  quantity  needed  to  address  the  critical 
issues.  Hardware  availability,  cost,  and  schedule  restrictions  make  it  difficult  to  achieve  all  test 
objectives.  But,  through  careful  test  planning,  including  forecasting  spare  parts  requirements, 
and  shot  sequencing,  test  processes  and  procedures  have  evolved  that  are  consistent  with,  and 
supportive  of,  the  critical  test  issues.  Addressing  critical  issues  in  the  planning,  execution,  and 
analysis  of  live-fire  tests  in  an  open  environment  with  adequate  oversight  by  all  levels  of  Service 
and  DoD  chains  of  command  has  improved  the  credibility  of  the  test  results  and  subsequent 
evaluations  significantly. 

LFT&E  has  been  beneficial  in  many  ways.  Specifically: 

•  Strengths  and  weaknesses  of  system  designs  have  been  identified  and  verified. 

Development  of  critical  issues  facilitates  identification  of  design  features  of  questionable 
robustness  and  the  development  of  a  strategy  to  investigate  these  features.  For  example, 
LFT  has  demonstrated  that  certain  munitions  were  incapable  of  perforating  threat  vehicle 
armor.  In  most  such  cases,  the  problem  was  rectified  through  changes  in  the  penetrator 
design.1-  But  not  all  news  about  system  performance  has  been  bad  news.  For  example,  it 
has  been  shown  repeatedly  that  not  all  impacts  by  overmatching  munitions  are  lethal  to 
either  the  system  or  crew.  In  fact,  the  probability  of  crew  casualties  or  system  loss  of 
combat  utility  has  often  been  shown  to  be  much  less  than  expected  prior  to  testing. 

Typically,  during  the  acquisition  program,  tests  are  conducted  to  verily  that  design 
requirements  have  been  met.  Historically,  these  tests  have  more  often  been  conducted  at 
the  component  or  subsystem  level  than  at  the  system  (platform)  level.  Nevertheless,  testing 


* 

In  a  lethality  FUSL  LFT  of  a  platform,  the  focus  is  the  terminal  effects  of  the  munition  on  the  target,  given  a  hit.  The  FUSL 
LFT  may  be  limited  to  tests  of  the  munition  and  be  totally  independent  of  the  identity  of  the  firing  platform. 

'  No  specific  systems  are  identified  in  order  to  keep  the  discussion  unclassified. 
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sought  to  assure  compliance  with  design  requirements.  These  tests  were  never  intended  to 
define  the  upper  limits  of  survivability  or  lethality.  LFT,  on  the  other  hand,  does  not 
directly  address  design  requirements.  Rather,  it  seeks  to  examine  survivability  and  lethality 
in  the  context  of  the  “full  spectrum  of  battlefield  threats,  to  include  ovennatching  threats 
(DA  Pam  73-1,  p.  203).”  Consequently,  testing  is  often  done  with  the  expectation  of 
significant  damage.  Testing  strives  to  determine  how  much  damage  occurs,  whether  the 
damage  is  expected,  whether  it  can  be  reasonably  mitigated,  and  whether  it  can  be  repaired 
using  Battle  Damage  Assessment  and  Repair  (BDAR)  practices.  This  has  produced 
valuable  information  to  the  designer  as  well  as  to  the  combat  user. 

•  Crew  hazards  from  the  spectrum  of  insults  have  been  quantified.  Prior  to  LFT,  only 
penetration  injuries  received  any  serious  analytical  consideration.  Injuries  from  blast, 
thermal  effects,  and  toxic  gases  were  either  ignored  or  feebly  accounted  for  by  adjusting 
component  kill  probabilities.  LFT  now  gives  considerable  attention  to  crew  hazards  and, 
as  a  result,  there  is  a  much  better  understanding  of  the  hazards  all  insults  really  pose  to  the 
crew  and  of  the  conditions  under  which  crew  injuries  can  be  expected. 

Overall,  penetration  injuries  remain  the  largest  contributor  of  crew  injuries.  However, 
testing  has  shown  that  other  insults  are  significant  threats  to  crew  survivability.  For 
example,  toxic  gases  from  burning  ammunition  and  other  combustibles  have  been 
identified  as  significant  hazards  if  no  protection  is  afforded  through  vehicle  or  individual 
protection.  With  protection,  the  hazard  is  greatly  mitigated.  Flash  blindness,  once  thought 
to  be  a  significant  hazard,  seldom  causes  serious  incapacitation.  Overpressure  from 
reactions  of  stowed  ammunition  can  be  a  serious  threat  to  crew  survivability.  Overpressure 
from  attacking  munitions  generally  does  not  affect  the  crew  of  heavy  fighting  vehicles,  but 
the  hazard  is  more  severe  for  crews  inside  light  vehicles. 

•  Weaknesses  and  shortcomings  in  analytical  models  have  been  identified.  Prior  to  the  LFT 
era,  even  so-called  high-resolution  component-level  models  did  not  have  the  resolution 
needed  to  investigate  specific  physics-of-interaction  phenomena  and  quantify  the  effects  of 
these  phenomena  at  the  component  level.  Those  models  were,  in  the  main,  expected  value 
models  that  predicted  a  point  estimate  of  the  probability  of  the  outcome  of  a  given  threat- 
target  interaction.  This  expected  value  was  often  interpreted  as  the  most  likely  outcome. 
The  models  neither  predicted  the  specific  outcome  of  a  particular  test  nor  gave  evidence  of 
the  range  of  outcomes  that  might  be  expected. 

LFT  motivated  a  long-term  investigation  of  not  only  the  quality  of  the  models  being  used, 
but  also  of  the  type  of  model  predictions  that  are  needed.  As  a  result,  models  now  predict 
distributions  of  possible  outcomes  rather  than  a  single  expected  value,  are  of  much  higher 
resolution  through  more  faithful  target  representation,  and  account  for  additional  damage 
mechanisms  and  crew  insults.  Much  more  work  is  needed  to  perfect  these  models  and, 
indeed,  much  is  ongoing.  Computation  architectures  are  being  developed  to  improve 
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efficiency  and  accommodate  improved  algorithms  for  computing  component  damage  and 
system  response.  Significant  resources  are  being  devoted  to  the  development  of  new 
algorithms  that  are  based  on  principles  of  physics  and  chemistry  with  less  dependence  on 
large  amounts  of  system-specific  empirical  data. 

LFT  has  motivated  true  analytical  model  validation.  Three  decades  ago,  critics  outside  the 
LFT  community  believed  the  only  true  indication  of  V/L  could  be  obtained  through 
destructive  testing.  They  believed  analytical  models  were  not  sufficiently  reliable  to  give  a 
true  picture  of  system  vulnerability  or  munition  lethality.  At  the  other  extreme,  a  few 
people  within  the  V/L  community  advocated  that  testing  was  not  only  unnecessary,  but  it 
was  wasteful  because  analytical  models  could  account  for,  and  quantify,  all  significant  V/L 
phenomena.  Now  nearly  everyone  recognizes  that  not  only  are  both  testing  and  analysis 
necessary,  they  can  and  must  be  complementary.  Analysis  guides  test  requirements  by 
identifying  areas  of  uncertainty  and  quantifying  the  consequences  of  particular  potential 
weaknesses  of  a  design.  Testing  provides  diagnostics  for  model  performance,  guides 
development  of  improved  algorithms,  and  supplies  data  with  which  the  models  can  be 
accurately  validated  at  the  algorithm  and  model  levels. 

Obviously,  all  threat-target  impact  conditions  cannot  be  tested.  Model  validation  derived  from 
comparisons  of  model  predictions  with  test  outcomes  during  the  last  two  decades  has  given  the 
V/L  community  confidence  that  models  can  be  used  to  extend  LFT  results  to  conditions  not 
tested.  Further,  the  models  have  been  used  to  limit  or  eliminate  shots  for  which  the  ability  of 
models  to  predict  the  outcome  has  been  demonstrated.  Such  proactive  use  of  V/L  models  has 
resulted  in  smarter  and  more  cost-effective  testing. 

1.3  Critical  Issues  of  LFT&E  Programs 

Since  the  onset  of  JLF  and  Title  10  LFT&E,  the  focus  has  been  directed  toward  defining  critical 
issues  associated  with  the  tested  platfonn  or  munition.  Typically,  critical  issues  in  vulnerability 
LFT&E  have  been  identified  as  follows: 


•  What  is  the  vulnerability  of  the  platfonn  against  the  spectrum  of  current  and  future  threats 
as  identified  by  the  intelligence  community? 

-  What  are  the  major  causes  of  crew  and  passenger  casualties? 

-  What  are  the  vulnerabilities  of  the  platform  to  perforating  and  nonperforating  threats? 

-  What  is  the  remaining  platform  mission  utility  after  the  shot? 

•  What  vulnerability  reduction  measures  are  effective  in  reducing  crew,  passenger,  and 
system  vulnerability? 

•  How  effective  is  BDAR  in  restoring  the  platform  to  functional  combat  capability  following 
an  attack? 
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Each  of  these  issues  is  generally  subdivided  into  several  sub-issues  relating  to  specific 
phenomena  of  interest,  for  example,  contribution  of  stowed  ammunition  to  casualties,  design 
features  of  the  specific  system  being  tested,  and  performance  of  specific  vulnerability  reduction 
designs  such  as  fire  suppression  hardware. 

In  the  context  of  the  V/L  taxonomy  (see  figure  2),*  these  issues  have  been  addressed  almost 
exclusively  at  Levels  2  (i.e.,  component  status)  and  4  (i.e.,  mission  utility).  Only  to  a  limited 
extent  have  results  been  related  to  Level  3  (i.e.,  capability  status).  Damage  assessment  results 
have  been  presented  to  identify  specific  components  and  subsystems  that  were  either  damaged  or 
proved  to  be  unexpectedly  robust.  From  this  infonnation,  it  has  been  possible  to  establish  design 
changes  that  were  needed  or  desired  and  the  consequences  of  not  implementing  those  changes. 
Results  for  armored  fighting  vehicles  have  also  been  presented  at  Level  4  with  the  aid  of  the 
damage  assessment  list  (DAL)f 


This  information,  cast  in  terms  of  probabilities  of  mission  or  catastrophic  kills,  allows  a 
generalization  of  the  results  and  examination  of  the  outcome  in  terms  of  a  global  ability  to 
complete  the  spectrum  of  missions  likely  to  be  assigned  to  the  platform. 

But,  while  remaining  mission  utility  has  been  represented  in  the  context  of  a  unit  assigned  to 
prosecute  a  generic  mission  intended  for  the  tested  platform,  the  platform  has  been  considered 


* 

The  taxonomy  of  the  V/L  analysis  process  was  first  introduced  in  Dcitz  and  Ozolins  (1989). 

^Similar  constructs  are  used  for  other  types  of  platforms,  including  aircraft.  LFT  results  can  be  presented  at  Level  4  only  with 
the  use  of  an  operations  research  mapping  format,  such  as  the  DAL.  Level  4  metrics  are  neither  observable  nor  testable  in  a 
laboratory  setting. 
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autonomous.  In  that  sense,  concern  has  focused  on  whether  the  damaged  platform  can  continue 
an  assigned  mission  or  perhaps,  through  BDAR,  be  used  for  an  alternate  mission.  Damage  has 
seldom  been  evaluated  in  the  context  of  specific  missions  or  complementing  capabilities  of  other 
platforms  within  the  mission-prosecuting  unit. 

Historically,  LFT  was  conceived  to  determine  survivability  and  lethality  of  an  autonomous 
combat  platform  in  a  combined  arms  team  operating  in  a  Cold  War  setting.  That  is,  battle 
doctrine  addressed  large-scale  confrontations  between  U.S.  and  U.S.S.R.  forces  engaging  in 
mostly  rural  areas.  While  the  concept  of  war  has  changed  in  recent  years,  the  modus  operandi  of 
LFT,  for  the  most  part,  has  not  changed.  It  still  focuses  on  determining  the  vulnerability  of  a 
single  platform  or  the  lethality  of  a  munition  fired  at  an  autonomous  platfonn.  LFT&E  programs 
for  recent  systems  of  interest  have,  to  be  sure,  recognized  that  platforms  no  longer  operate  as 
autonomous  entities  because  platforms  and  even  units  are  necessarily  linked  by  vast  amounts  of 
real-time  information  that  directs  the  conduct  of  even  individual  battles.  But,  so  far,  LFT 
strategies  have  chosen  to  address  the  linkage  between  individual  platforms  and  their  units 
through  force-on-force  M&S  and  not  through  the  LFT  planning  or  damage  assessment  processes. 

As  valuable  as  LFT&E  has  turned  out  to  be,  the  design  and  conduct  of  LFT  and  the  use  of  data 
produced  by  that  testing  need  to  be  changed  to  better  evaluate  the  subject  platfonn  in  the  System 
of  Systems  (SoS)  context.  Changes  in  the  kinds  of  wars  fought,  the  kinds  of  systems  employed, 
the  use  of  these  systems  on  the  integrated  and  information-centric  battlefield,  and  the  evolving 
nature  of  the  acquisition  process  demand  a  fresh  look  at  LFT  to  make  it  more  relevant  and  useful 
in  the  future. 

It  is  the  premise  of  this  report  that  LFT&E  must  not  operate  as  an  isolated  activity  during  system 
acquisition.  The  LFT&E  program  must  reflect  the  role  of  the  tested  platform  in  an  SoS  construct 
and  address  individual  platform  capabilities  in  the  context  of  the  collection  of  platform 
capabilities  available  to  the  unit  commander  to  prosecute  a  wide  range  of  missions.  Further, 
while  actual  ballistic  testing  will  generally  involve  a  single  platform,  damage  assessment  can  be 
expanded  to  experimentally  evaluate  remaining  capabilities  of  the  damaged  platform  in  a  unit 
setting  through  select  quasi-operational  tests. 

Live-Fire  Testing  and,  most  importantly,  Evaluation,  must  be  considered  in  the  context  of  SoS 
constructs  where  individual  platforms  are  linked  by  sophisticated  infonnation  networks.  Critical 
issues  should  reflect  platform  capabilities  that  are  required  to  accomplish  a  mission  by  a 
collection  of  platforms.  It  may  turn  out  that  a  platform  capability  critical  for  mission 
performance  in  the  context  of  autonomous  operations  can  be  provided  by  a  companion  platform 
when  the  small  unit  is  considered.  The  consequence  of  developing  critical  issues  from  this  point- 
of-view  is  that  shot  selection  may  be  different  than  that  appropriate  for  an  autonomous  platform. 

Damage  assessment  has  traditionally  cataloged  damage  inflicted  in  an  LFT  and  examined  the 
effects  of  that  damage  in  limiting  capabilities  of  that  platfonn.  For  example,  damage  to  a  sight  is 
evaluated  in  terms  of  reducing  the  firepower  of  the  vehicle.  Considering  LFT&E  from  an  SoS 
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perspective  requires  the  damage  to  be  evaluated  in  terms  of  how  the  mission  completion 
capability  of  the  entire  unit  will  be  affected.  One  method  of  doing  this  is  through  quasi- 
operational  testing  where  the  damaged  vehicle  can  operate  with  companion  vehicles  to  detennine 
the  true  effect  of  any  subsystem  functional  degradation.  Obviously,  not  all  test  outcomes  can  be 
assessed  in  this  manner.  Resources  will  allow  only  a  small  sampling  of  numerous  possibilities, 
but  even  some  such  tests  will  shed  light  onto  our  ability  to  predict  the  extent  to  which  missions 
will  be  jeopardized  by  damage  to  given  platforms. 

1.4  Report  Objectives 

The  major  objective  in  developing  an  effective  Test  and  Evaluation  (T&E)  plan  for  a  platfonn  that 
is  part  of  a  complex  SoS  is  to  facilitate  the  measurement  and  assessment  of  (1)  the  effectiveness, 
suitability,  and  survivability  of  the  platfonn  relevant  to  its  contributions  to  the  SoS  and  (2)  the 
effectiveness  of  the  SoS  in  accomplishing  the  assigned  operational  missions  in  the  joint  operational 
enviromnent.  This  report  describes  a  methodology  for  constructing  cost-effective  programs  for 
LFT&E,  a  major  component  of  the  T&E  plan  and  the  V/L  assessment  strategy,  within  the 
Missions  and  Means  Framework  (MMF)  environment. 

The  report  addresses  the  following  issues: 

•  How  should  the  processes  for  conducting  LFT&E  change  in  light  of  the  tactical 
considerations  of  SoS  combat  doctrines? 

•  How  should  the  LFT&E  program  be  designed  and  conducted  to  ensure  (1)  the  collection  of 
data  relevant  to  V/L  assessment  decisions,  (2)  the  generation  of  the  optimal  combination  of 
component-level,  subsystem-level,  and  system-level  test  data,  with  consideration  given  to 
cost,  availability  of  hardware,  and  production  schedule,  and  (3)  the  evaluation  of  LFT 
results  in  a  format  that  is  useful  to  decision-makers  concerned  with  accomplishing  SoS 
collective  tasks  and  achieving  mission  success  in  the  joint  operating  environment? 


2.  Missions  and  Means  Framework 


The  MMF  provides  a  structure  for  “specifying  the  military  mission  and  quantitatively  evaluating 
the  mission  utility  of  alternative  warfighting  Doctrine,  Organization,  Training,  Materiel,  Leader 
Development,  Personnel,  and  Facilities  (DOTMLPF)  services  and  products  (Sheehan  et  ah, 
2004b).”  It  serves  as  a  bridge  from  the  warfighter  operational  community  to  the  warfighter 
support  communities  (see  figure  3). 

The  framework  supplies  a  disciplined  procedure  for  explicitly  specifying  the  mission,  allocating 
the  means,  and  assessing  the  accomplishment  of  the  mission.  A  brief  description  of  the 
components  of  the  MMF  in  the  interaction  of  friendly  (OWNFOR)  and  opposing  (OPFOR) 
forces  follows. 
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MMF  Connections 


>  Operations  Research  and  Systems  Analysis 
>Materiel  Acquisition  and  Force  Development 

>  Logistical  Support  and  Life-Cycle  Sustainment 
/'Concept  Analysis,  Cost  Analysis 

>  Modeling  and  Simulation 

>  Requirements  Determination 

>  Training  and  Readiness 

>  Vulnerability /Lethality  Assessment 
>Test  and  Evaluation 


Warfighter  Support 
Communities 


Figure  3.  MMF  links  between  warfighter  operational  and  support  communities. 

Figure  4  represents  the  activity  of  a  mission  under  conditions  created  by  location  and  time  of 
mission  performance.  Specified,  implied,  and  essential  tasks,  enabled  by  capabilities,  are 
performed  by  task-organized  units.  The  MMF  is  composed  of  seven  levels  including: 

7.  the  mission — the  assignment  that  indicates  the  purpose  of  actions  to  be  taken  and  the 
required  outcomes; 

6.  the  environment — the  military,  civil,  and  physical  context  and  unit-specific  conditions  that 
explain  the  circumstances  under  which  actions  will  be  taken  in  the  accomplishment  of  the  mission; 

5.  the  location  and  time — the  index  that  describes  the  when  and  where  of  actions  to  be  taken; 

4.  the  tasks  and  operations  — the  activity  components  that  are  needed  to  accomplish  the 
mission; 

3.  the  functions  and  capabilities — the  abilities  to  move,  sense,  communicate,  engage,  etc., 
that  are  required  for  task  completion; 

2.  the  components  of  forces — the  networks  of  units,  personnel,  and  equipment  that  provide 
the  capabilities  to  complete  the  tasks;  and 

1 .  the  interactions — the  actions  that  result  in  changes  to  the  components  of  forces  through 
the  processes  of  mission  execution  (i.e.,  normal  wear  and  tear  and  damage  from  hostile  forces). 
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Figure  4.  Two-sided  MMF.* 

The  MMF  also  includes  four  operators,  illustrated  in  figure  4  by  the  pairs  of  arrows  that  connect 
the  four  levels  of  (1)  interactions/effects,  (2)  components/forces,  (3)  functions/capabilities,  and  (4) 
tasks/operations/mission.  Beginning  with  OWNFOR  tasks  and  proceeding  in  a  counter-clockwise 
direction  (top-down  direction)'1'  toward  capabilities,  the  blue  arrows  indicate  the  concurrent 
synthesis  and  decision-making  process;  causal,  time-forward  execution  and  adjudication  of 
outcomes  is  explained  by  proceeding  in  a  clockwise  direction  (bottom-up  direction)11  from 
interactions  and  effects  toward  tasks  and  operations. 

Mission-task  decomposition,  an  important  component  of  the  mission  analysis  process  explained  in 
the  preceding  paragraphs,  describes  the  tasks  that  must  be  completed  to  accomplish  the  mission. 
Tasks  are  derived  from  the  Universal  Joint  Task  List  (UJTL),-  the  Service  Task  Lists  (i.e.,  Army 
Universal  Task  List  [AUTL],  Universal  Navy  Task  List  [UNTL],  and  Air  Force  Task  List 
[AFTL]),  and  Mission  Training  Plans  (MTPs)  and  associated  Operations  Plans  (OPLANs)  or 
Operations  Orders  (OPORDs).  Task  lists  define  both  conditions  (physical,  military,  and  civil 
environment)  and  measures  of  performances  (i.e.,  measure,  scale,  and  criterion-standard)  for  task 
completion. 


Figure  adapated  from  Sheehan  et  al.  (2004a).  OPFOR  movements  would  be  the  opposite  of  OWNFOR  movements  (i.e., 
OWNFOR  counter-clockwise  would  be  OPFOR  clockwise  and  OWNFOR  clockwise  would  be  OPFOR  counter-clockwise). 

^Clockwise  direction  (bottom-up  direction)  for  OPFOR  description. 

■^Counter-clockwise  direction  (top-down  direction)  for  OPFOR  description. 

^Joint  operations  require  the  use  of  UJTL  tasks.  The  UJTL  lists  tasks  in  a  hierarchical  manner,  identifying  what  is  to  be 
performed  by  Joint  forces,  under  Joint  command,  using  Joint  doctrine.  CJCSM  3500-04C,  Universal  Joint  Task  List  (2002),  states 
that  the  UJTL  “serves  as  a  common  language  and  common  reference  system  for  joint  force  commanders,  combat  support 
agencies,  operational  planners,  combat  developers  and  trainers  to  communicate  mission  requirements.” 
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The  MMF  provides  a  layered  perspective,  identifying  missions  and  tasks  by  the  levels  of  war. 

For  example,  as  illustrated  in  figure  5,  the  Strategic  National  level’s  mission  to  protect  national 
interests  might  be  composed  of  tasks  to  restore  the  legitimate  government  (e.g.,  employ  forces, 
decide  on  need  for  military  action)  and  require  national  intelligence  and  communications  assets. 
A  Strategic  Theatre  level  that  assumes  the  mission  to  restore  the  legitimate  government  may 
complete  tasks  to  establish  certain  military  and  civil  conditions  (e.g.,  conduct  operations  in 
depth)  and  employ  a  joint  task  force  to  provide  the  necessary  capabilities.  The  mission  of  the 
Operational  level  to  help  establish  the  desired  military  and  civil  conditions  may  require  the 
completion  of  tasks  to  isolate  the  rebel  govermnent  (e.g.,  conduct  offensive  operations)  and 
utilize  air,  ground,  maritime,  and  special  operation  forces  elements.  The  Tactical-Joint  level  may 
employ  a  U.S.  Army  Future  Combat  System  (FCS)-equipped  Brigade  Combat  Team  (BCT)  to 
complete  tasks  to  prevent  the  reinforcement  of  the  enemy’s  capital  (e.g.,  conduct  an  attack)  in  an 
attempt  to  accomplish  its  mission  to  help  isolate  the  rebel  government.  The  Tactical-Service 
level,  given  the  mission  of  preventing  the  reinforcement  of  the  capital  by  accomplishing  tasks 
focused  on  blocking  the  access  on  the  main  route  into  the  capital  (e.g.,  seize  a  specific  area)  may 
choose  to  employ  a  combined  arms  battalion  (CAB).*  As  this  brief  example  illustrates,  the 
mission  and  its  task  components,  as  well  as  the  capabilities  and  the  force  assets  required  to 
provide  the  identified  capabilities,  differ  with  the  layer  or  level  of  war  considered. 

The  task  organization  process  provides  a  basis  for  the  identification  of  the  force  assets  to  best 
achieve  the  mission  goals  by  linking  (1)  the  capabilities  to  tasks,  given  the  actual  or  anticipated 
conditions  and  standards,  and  (2)  the  capabilities  to  the  resources  available.  The  difference 
(i.e.,  delta)  between  the  capabilities  required  and  the  capabilities  supplied  by  available  force 
assets  is  addressed  with  a  recommended  solution  in  one  or  more  of  the  DOTMLPF  areas  (Bray, 
2005). 


3.  The  MMF  and  LFT&E  in  SoS  Environment 


Linking  missions  and  means  is  vitally  important  to  a  multitude  of  defense/military  efforts,1' 
including  the  development  of  combat  strategies,  the  preparation  of  an  analysis  of  alternatives 
(AoA),  the  development  of  training  programs,  the  development  of  M&S  tools,  system 
development  and  acquisition,  the  development  of  study,  experimentation  and  test  plans,  the 
assessment  of  readiness,  the  estimation  of  mission  costs,  and  the  planning,  conduct,  and 
evaluation  of  developmental,  operational,  and  LFT  programs  and  their  results.  All  of  the 
aforementioned  efforts  must  be  considered  within  the  complex  SoS  environment. 


* 

The  UJTL  defines  tasks  at  strategic  and  operational  levels,  and  the  Service-specific  task  lists  define  the  tactical-Service  tasks. 

^The  MMF  supports  the  Joint  Capability  Integration  and  Development  System,  one  of  the  three  principal  decision  support 
processes  for  transforming  the  military  forces  according  to  the  future  DoD  vision  (Tanenbaum  and  Bray,  2005). 
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Strategic  National  Level  Mission:  Protect  national  interests 
Tasks: 

1.  Restore  legitimate  government  of  Jland 

2.  Achieve  air  superiority 

3.  Destroy  weapons  of  mass  destruction  capability 

4.  Facilitate  control  of  energy  centers 


Strategic  Theatre  Level  Mission:  Restore  legitimate  government  of  Jland 
Tasks: 

1.  Secure  capital  region 

2.  Defeat  rebel  forces 

3.  Secure  key  oil  and  gas  pipelines  and  production  facilities 


'Operational  Level  Mission:  Defeat  rebel  forces;  secure  capital  region 
Tasks: 

1.  Attack  to  defeat  rebel  forces  in  zone 
'2.  Conduct  vertical  maneuver  to  disrupt  rebel  forces 

3.  Secure  key  facilities 


Tactical-Joint  Level  Mission:  Help  to  isolate  rebel  government  by 

disrupting  and  defeating  rebel  forces 

Tasks: 

1 .  Secure  key  terrain 

2.  Block  enemy  reinforcements  from  reaching  capital _ 


Tactical-Service  Level  Mission:  Prevent  reinforcement  of  rebel  forces 
Tasks: 

1.  Block  access  on  road  to  capital 

2.  Attack  to  defeat  rebel  forces  attempting  to  return  to  capital 


Figure  5.  Strategic  national  to  tactical  nested  tasks.* 


Within  the  SoS  environment,  the  objective  of  T&E  is  to  facilitate  the  measurement  and 
assessment  of  (1)  the  effectiveness,  suitability,  and  survivability  of  platforms  relevant  to  their 
contributions  to  the  SoS  and  (2)  the  effectiveness  of  the  SoS  in  achieving  mission  objectives  at 
the  Joint  Forces  Command  (JFC)  level.  T&E,  therefore,  requires  the  assessment  of  the 
capabilities  of  SoS  platforms,  both  individually  and  collectively,  to  complete  identified  tasks  in 
tactically  realistic  scenarios.  The  assessment  of  task  performance  requires  an  understanding  of 
task  sub-parts,  the  standards  of  performance  expected,  and  the  conditions  under  which  the  tasks 
may  be  performed,  including  the  impact  of  those  conditions  on  task  performance.  The  rolling  up 
of  T&E  results  from  the  platform/SoS  level  to  the  JFC  level  requires  an  understanding  of  the 
mission  hierarchy  that  induces  tasks,  conditions,  and  standards,  as  well  as  an  understanding  of 
the  hardware  hierarchy(ies)  that  induces  capabilities. 


* 

Figure  modified  from  Sheehan  et  al.  (2004b). 
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As  illustrated  in  figure  6,  the  gap  between  the  two  identified  hierarchies  (i.e.,  Tasks  and 
Capabilities)  leads  to  the  question,  “Do  we  have  enough  capability  to  complete  tasks  to  standard 
under  the  given  conditions?”  or  “Does  the  mission  capability  package  meet  the  mission 
capability  requirement?”  To  address  these  questions,  decision-makers  must  consider  the 
acceptable  levels  of  risk  of  failure  in  completing  the  mission,  as  well  as  alternative  courses  of 
SoS  action  (Tanenbaum  and  Bray,  2005).  The  vulnerability  and  lethality  of  platforms  and  their 
munitions  in  ballistic  interactions,*  assessed  in  LFT&E  programs,  are  important  factors  in  this 
decision  process. 


3.1  Cost-Effective  V/L  Assessment  in  SoS  Environment 

In  the  traditional  platform-centric  view  of  vulnerability  assessment,  attention  is  directed  toward 
threat-system  interactions  that  degrade  the  system’s  capabilities  or  that  result  in  injuries  to  the 
system’s  crewmembers  and  passengers.  The  question  addressed  is,  “To  what  extent  will  the 
interactions  of  the  weapon  system  and  the  threats  that  the  system  is  likely  to  encounter  in  combat 
result  in  personnel  casualties  (i.e.,  personnel  vulnerability)  or  the  loss  of  the  system’s  capabilities 
(i.e.,  system  vulnerability)?” 


Although  some  programs  consider  certain  conventional  nonballistic  threats  (e.g.,  lasers,  high-powered  microwaves),  it  is 
assumed  in  this  report  that  the  focus  of  LFT&E  programs  is  ballistic  interactions. 

'Figure  modified  from  Tanenbaum  and  Bray  (2005). 
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In  a  similar  manner,  lethality  assessment  has  been  historically  viewed  as  a  process  by  which 
decision-makers  ascertain  the  extent  to  which  the  interactions  of  the  system  and  the  threat 
platform  (i.e.,  opposition’s  platform)  eliminate  or  degrade  the  operational  functions  of  the  threat 
platform,  resulting  in  the  reduction  of  the  levels  of  the  threat  platform’s  capabilities.  This 
includes  damage  to  or  catastrophic  loss  of  the  threat  platform,  as  well  as  casualties  that  render 
threat  the  platform’s  crew  unable  to  complete  the  mission  tasks  of  the  opposition.* 

How  does  the  process  for  conducting  V/L  assessment  and  LFT&E  change  in  light  of  the  tactical 
considerations  of  SoS  combat  doctrines?  First,  the  significance  of  threats  on  the  battlefield  can 
be  assessed  in  terms  of  their  potential  impact  on  the  SoS  rather  than  their  impact  on  a  single 
platform.  Regional  threats  and  widely  proliferated  threats  may  pose  a  greater  hazard  to  a  unit’s 
ability  to  perfonn  its  mission  than  a  less  common,  but  more  damaging  threat.  For  example,  a 
munition  that  disrupts  communications  among  multiple  platfonns  could  impact  a  unit’s  assigned 
mission  more  severely  than  the  catastrophic  loss  of  a  single  platform  to  a  severe  ovennatch. 
Second,  in  the  SoS  environment  with  the  emphasis  on  joint  operations,  it  is  important  to  extend 
the  focus  of  vulnerability  (lethality)  assessment  beyond  the  evaluation  of  the  functional 
capabilities  of  the  platform  (threat  platform)  and  include  the  assessment  of  the  capabilities  of  the 
SoS  and  subsequent  prosecution  of  the  mission  at  the  JFC  level. 

It  is  proposed  that  in  an  effective  vulnerability  LFT&E  program  in  an  SoS  environment, 
evaluators  must  assess  (1)  the  extent  to  which  the  weapon  system  retains  those  capabilities 
determined  (at  time  of  acquisition)  to  be  needed  for  completion  of  SoS  tasks,  when  the  system 
interacts  with  the  full  spectrum  of  ballistic  threats  it  is  likely  to  encounter  in  combat,  and  (2)  the 
extent  to  which  the  SoS  is  able  to  complete  the  identified  mission  tasks  in  the  joint  environment, 
given  the  residual  and  available  (i.e.,  as  determined  in  LFT&E)  capabilities  of  the  tested 
platform.  Likewise,  an  effective  lethality  LFT&E  program  evaluates  (1)  the  extent  to  which  a 
threat  platform  retains  the  capabilities  needed  for  completion  of  its  mission  tasks  following 
ballistic  interaction  with  the  assessed  munitions  and  (2)  the  extent  to  which  the  SoS  is  able  to 
complete  the  identified  mission  tasks  in  the  joint  environment,  given  the  results  of  the  lethality 
tests  (see  figure  7). 

To  make  V/L  assessment  and  LFT&E  not  only  effective,  but  also  cost-effective,  budgetary 
constraints  must  be  considered.  To  address  the  question  posed  in  figure  7,  V/L  assessment 
strategies  and  LFT&E  programs  must  be  efficient  and  designed  with  an  eye  toward  the  final 
objective:  the  assessment  of  the  ability  of  the  platform  to  prosecute  the  mission  as  a  component 
of  an  SoS  in  the  joint  environment. 


* 

The  word  “lethality”  is  often  used  in  a  more  narrow  sense  with  the  focus  solely  on  damage  to  the  target  platform,  and  the 
tenn  “effectiveness”  is  used  to  capture  the  damage  and  consequences  of  that  damage. 
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Figure  7.  Impacts  of  Level  2  component  state  changes  on  Level  4  mission. 

To  design  a  cost-effective  LFT&E  program,  planners  must  ascertain  what  data  are  required  by 
system  evaluators  for  assessment  decisions  (i.e.,  required  data  set)  relative  to  ballistic  interactions 
and  compare  that  required  data  set  to  the  subset  of  reliable  and  relevant  data  available.  The  subset 
of  data  required  for  assessment  that  is  unavailable  or  unable  to  be  relied  upon  is  identified  as  the 
data  voids.  With  consideration  given  to  the  limitations  in  available  resources,  the  data  voids  must 
be  prioritized  and  addressed  in  the  design  of  the  LFT&E  program  (Nelson,  2000). 

The  MMF  provides  the  foundation  for  identifying  the  required  data  set  and  data  voids  to  be 
addressed  in  elements  of  the  LFT&E  program.  Specific  to  LFT&E,  MMF  provides  the  basis  for 
(1)  the  identification  of  critical  issues  to  be  addressed  in  LFT&E,  (2)  the  design  of  LFT  programs 
to  address  prioritized  data  voids,  and  (3)  the  design  and  execution  of  the  evaluation  process,  in 
which  the  results  of  LFT  are  considered  along  with  the  results  of  other  program  supporting 
activities  (i.e.,  M&S,  CDE,  etc.)  with  the  objective  of  determining  the  consequences  of  the 
effects  of  ballistic  interactions  on  mission  tasks  and  related  DOTMLPF.  The  MMF  plays  an 
important  role  in  the  LFT&E  activities  found  in  the  shaded  areas  of  figure  8.  The  following 
section  provides  a  comparison  of  traditional  platform-centric  LFT&E  strategies  and  MMF-based 
SoS  task- focused  LFT&E  strategies. 


* 

Figure  modified  from  Tanenbaum  and  Bray  (2005). 
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Figure  8.  LFT&E  activities  in  which  MMF  plays  an  important  role. 

3.2  Comparison  of  Traditional  and  MMF-Based  LFT&E  Strategies 

Table  1  illustrates  three  major  areas  in  which  the  traditional  platform-centric  LFT&E  strategy  is 
expected  to  differ  from  the  proposed  MMF-based  SoS  task-focused  LFT&E  strategy:  Critical 
Issues,  Shotline  Selection,  and  Damage  Assessment. 

3.2.1  Critical  Issues 

Typically,  traditional  vulnerability  platform-centric  strategies  have  addressed  three  critical 
issues:  (1)  the  vulnerability  of  the  platform  and  crew/passengers  to  current  and  expected  future 
perforating  and  nonperforating  threats,  (2)  the  effectiveness  of  vulnerability  reduction  measures, 
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Table  1.  Comparison  of  traditional  and  MMF-based  LFT&E  strategies. 


Traditional  Platform-Centric 

LFT&E  Strategy 

MMF-Based  SoS  Task-Focused  LFT&E 
Strategy 

Critical 

Issues 

What  is  the  vulnerability  of  the  platform  to 

current  and  future  threats  identified  by  the 
intelligence  community? 

•  What  are  the  major  causes  of  crew  and  passenger 
casualties? 

•  What  are  the  vulnerabilities  of  the  platform  to 
perforating  and  non-perforating  threats? 

•  What  is  the  remaining  platform  mission  utility 
after  the  shot? 

What  is  the  reduction  in  the  ability  of  the  SoS  to 
prosecute  typical*  missions  after  damage  from 

current  and  future  threats  identified  by  the 
intelligence  community? 

•  What  are  the  major  causes  of  crew  and  passenger 
casualties? 

•  What  are  the  vulnerabilities  of  the  platform  to 
perforating  and  non-perforating  threats? 

•  What  are  the  remaining  platform  capabilities  after 
the  shot? 

What  vulnerability  reduction  measures  are 
effective  in  reducing  crew,  passenger,  and  system 
vulnerability? 

What  vulnerability  reduction  measures  are  effective  in 
reducing  crew,  passenger,  and  system  vulnerability? 

Flow  effective  is  BDAR  in  restoring  the  platform 
to  functional  capability  after  an  attack? 

Flow  effective  are  BDAR  and  other  maintenance 
actions  in  restoring  SoS  capabilities  critical  to 
mission  prosecution  after  an  attack? 

•S  -2 

o 
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Platform:  based  on  technical  risk  associated  with  the 
inability  to  determine  platform  capability  as  the 
result  of  ballistic  damage. 

Platform:  based  on  technical  risk  associated  with  the 
inability  to  determine  the  effect  on  mission 
prosecution  caused  by  loss  of  platform  capabilities 
as  the  result  of  ballistic  damage. 

QJ 
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Crew:  based  on  technical  risk  associated  with  the 
inability  to  predict  injury  to  crew  and  passengers. 

Crew:  based  on  technical  risk  associated  with  the 
inability  to  predict  injury  to  crew  and  passengers 

Platform: 

•  Determine  damage  to  components  and 
subsystem  loss  of  function. 

•  Map  to  remaining  combat  utility  via  DAL  or 
other  034  construct. 

Platform: 

•  Determine  damage  to  components  and  subsystem 
loss  of  function. 

•  Map  to  remaining  SoS  capabilities  by  analysis 
and  operational-type  tests. 

Damage 

Assessment 

Crew: 

•  Determine  casualties  among  crew  and  passengers. 

•  Map  crew  incapacitation  to  remaining  combat 
utility  via  DAL  or  other  03,4  construct. 

Crew: 

•  Determine  casualties  among  crew  and  passengers. 

•  Map  crew  incapacitation  to  platform  loss  of 
capabilities  and  confirm  remaining  SoS 
capabilities  by  analysis  and  operational-type 
tests. 

BDAR:  Determine  expedient  repairs  that  can  be 
made  to  restore  platform  to  some  level  of  combat 
utility. 

Mission  Damage  Assessment  and  Repair: 

BDAR:  Determine  expedient  repairs  that  can  be  made 
to  restore  some  platform  capabilities  during  and 
immediately  following  an  engagement. 

Other  maintenance  procedures:  Conduct  further 
repair  to  anticipate  future  mission  engagements. 

*  . 

LFT&E  considers  all  known  and  anticipated  threats  and  uses  of  the  platfonn.  Unable  to  examine  all  scenarios,  LFT&E 
focuses  on  those  most  likely  and  those  of  high  interest  to  decision-makers.  One  outcome  of  this  is  that  threats  likely  to  be 
encountered  now  or  in  the  future  in  the  selected  scenarios  can  be  emphasized.  Currently,  the  LFT&E  strategies  tend  to  consider 
all  threats  that  are  likely  to  be  encountered  in  any  scenario.  The  MMF  provides  a  basis  for  scenario  prioritization  in  the 
identification  of  critical  issues. 
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and  (3)  the  effectiveness  of  BDAR  in  platform  restoration.  An  MMF -based  SoS  task-focused 
strategy  addresses  the  same  three  critical  issues,  but  from  a  different  perspective  than  that  of  the 
traditional  platform-centric  strategy. 

The  traditional  platform-centric  strategy  focuses  on  the  extent  to  which  a  platform  retains 
battlefield  combat  utility  (Level  4)  or  a  general  ability  to  complete  missions  likely  to  be  assigned 
to  the  platform  when  subjected  to  current  and  expected  future  threats.  The  platform  is 
considered  autonomous,  and  little  consideration  is  given  in  the  critical  issues  to  the 
complementary  capabilities  of  other  platforms  that  are  part  of  the  SoS  or  to  the  platform’s  role  in 
the  completion  of  tasks  that  are  linked  to  specific  missions  in  the  joint  environment. 

In  contrast,  the  MMF-based  SoS  task-focused  strategy  focuses  on  the  capabilities  of  the  platform 
within  the  context  of  the  SoS  (Level  3)  and  the  extent  to  which  damage  to  the  platform  from 
current  and  expected  future  threats  affects  the  SoS’s  ability  (Level  3)  to  complete  specific 
mission  tasks  (Level  4).*  Redundancies  and  interdependencies  among  SoS  platforms  are 
considered  in  the  identification  of  critical  issues  and  prioritization  of  data  voids. 

In  principle,  there  is  no  difference  between  the  two  strategies  in  their  approach  to  identifying  the 
vulnerabilities  of  the  platform  to  perforating  and  nonperforating  threats.  The  platfonn  is  the 
most  basic  element  of  an  SoS  from  an  LFT  perspective.  Thus,  it  is  necessary  to  understand  and 
quantify  the  effects  of  damage  on  component  operation  and  subsystem  function  whether  the 
focus  is  the  single  platform  or  the  SoS  and  its  mission  tasks.  The  technical  risk  associated  with 
the  uncertainty  of  the  threat  effects  on  some  components  and  subsystems,  however,  may  be 
different  for  the  MMF-based  SoS  task- focused  strategy  with  its  SoS  perspective  (see  Shotline 
Selection,  section  3.2.2). 

Both  strategies  are  concerned  with  crew  and  passenger  survivability,  which  encompasses  the 
susceptibility  of  the  crew  and  passengers  to  injury  from  all  insults.1'  Personnel  survivability 
commands  high  priority  regardless  of  mission  considerations,  and  both  strategies  consider  the 
identification  of  the  major  causes  of  crew  and  passenger  casualties  and  the  evaluation  of  the 
effectiveness  of  personnel  protection  measures  as  critical  issues. 

In  a  traditional  platfonn-centric  strategy,  the  LFT&E  BDAR  focus  is  on  restoring  as  many 
platform  capabilities  as  possible  to  give  the  commander  further  use  of  the  vehicle  in  its  intended 
or  alternate  role,  depending  on  damage.  For  example,  LFT  of  combat  platfonns  has  generally 
been  conducted  in  the  context  of  a  10  or  15  minute  firelight. 1  BDAR  assessments  have  been 


* 

The  MMF  links  tasks  of  specific  missions  (Level  4)  to  the  capabilities  (Level  3)  that  are  required  to  complete  those  tasks.  A 
task  may  be  completed  by  a  single  platfonn  or  a  group  of  platfonns. 

fyrom  an  LFT  perspective,  this  often  involves  armor  protection  and  fire/explosion  if  there  are  on-board  combustibles. 

^Traditionally,  Anny  annored  fighting  vehicles  (AFVs)  in  close  combat  were  analyzed  in  terns  of  their  ability  to  continue  or 
escape  from  an  engagement  lasting  10  or  15  minutes.  Other  timeframes  have  been  used  for  other  types  of  engagements 
(e.g.,  interdiction  missions)  and  for  other  types  of  ground  mobile  vehicles,  such  as  trucks  and  air  defense  systems.  Army 
helicopters  were  analyzed  in  tenns  of  their  ability  to  continue  controlled  flight  long  enough  to  land,  return  to  base,  or  continue  the 
mission. 
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performed  to  determine  whether  the  crew  could  effect  sufficient  repairs  to  continue  the  mission, 
to  escape,  or  to  make  the  platfonn  available  for  an  alternate  mission  through  restoration  of  some 
of  its  capabilities. 

With  its  emphasis  on  SoS  mission  tasks,  the  mission  damage  assessment  and  repair  of  the 
MMF-based  SoS  task- focused  strategy  may  extend  beyond  the  activities  associated  with 
traditional  platform-centric  BDAR  to  include  a  more  extensive  plan  for  assessing  and  repairing 
battle  damage  and  other  failures.  For  longer  duration  complex  missions  in  which  the  platform 
may  be  involved  in  several  engagements,  conventional  repair  processes  may  be  available  to 
effect  more  complex  repairs,  restore  platform  capabilities  needed  later  in  the  mission,  and  even 
repair  reliability  failures  or  nonballistic  combat  damage  (see  section  3.2.3  for  additional 
discussion). 

3.2.2  Shotline  Selection 

Shotlines  in  an  LFT&E  program  are  selected  with  the  objective  of  minimizing  the  technical  risk 
associated  with  the  inability  to  predict  platform  capability  and  crew/passenger  casualties 
following  ballistic  interactions.  Shotline  selection  may  differ  between  the  traditional  platform¬ 
centric  and  MMF-based  SoS  task- focused  strategies  in  cases  where  data  voids  are  related  to  the 
platform  (vs.  data  voids  related  to  personnel  safety).*  Shotline  selection  differences  would 
reflect  the  distinction  between  the  MMF-based  SoS  task-focused  strategy  focus  on  SoS 
capabilities  and  the  traditional  platform-centric  strategy  focus  on  the  functional  capability  of  the 
autonomous  platform. 

With  limited  time  and  dollars  to  devote  to  LFT,  shotline  selection  requires  the  prioritization  of 
identified  data  voids — areas  in  which  there  is  little  understanding  of  the  effects  of  ballistic 
interactions  on  platform  capabilities.  In  the  MMF-based  SoS  task-focused  strategy,  the  limited 
LFT  shots  must  be  “spent”  to  address  the  most  urgent  questions  of  a  platform’ s  ability  to  support 
the  SoS  mission  tasks.  The  technical  risk  associated  with  failing  to  address  less  critical 
capabilities  in  an  LFT  may  be  acceptable,  however,  particularly  if  some  data  are  available  from 
component  tests  or  other  sources. 

Consider  the  following  examples  of  two  platforms,  each  with  significant  firepower  capability — an 
armored  personnel  carrier  (APC)  equipped  with  an  autocannon  and  a  main  battle  tank  (MBT).  The 
role  of  the  autocannon  on  the  APC  is  to  provide  self-defense  and  covering  fire  for  dismounted 
troops  and  to  attack  targets  of  opportunity.  From  the  perspective  of  the  commander  assigned 
typical  missions,  the  APC’s  firepower  may  be  of  secondary  importance,  because  the  primary 
capability  of  the  APC  is  mobility  for  troop  transport.  The  APC  would  generally  operate  with  other 
platforms  in  accomplishing  its  primary  role,  and  any  necessary  firepower  could  be  provided  by 
companion  vehicles. 


All  LFT&E  strategies  would  be  expected  to  give  a  high  priority  to  the  investigation  of  data  voids  relative  to  potential  crew 
and  passenger  casualties. 
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In  both  the  traditional  platform-centric  and  MMF-based  SoS  task-focused  strategies,  critical 
issues  would  include  the  vulnerability  of  the  autocannon,  related  fire  control,  target  acquisition, 
and  ammunition  handling  subsystems.  In  the  MMF-based  SoS  task- focused  strategies,  however, 
the  technical  risk  associated  with  relying  on  modeling  or  previously  generated  and  partially 
applicable  data  from  other  systems  relative  to  firepower  issues  may  be  acceptable.  Devoting 
precious  LFT  shots  to  mobility  and  crew  survivability  issues,  which  are  of  more  interest  from  a 
mission  prosecution  perspective,  might  be  a  better  investment  and,  overall,  of  less  technical  risk. 

In  contrast,  the  tendency  in  traditional  platform-centric  strategies  would  be  to  devote  some  shots 
to  the  firepower  capability  because  the  critical  sub-issues  tend  to  consider  all  platform 
capabilities  equally  important.* 

In  the  second  example,  the  MBT,  however,  not  much  difference  would  be  expected  between  the 
two  strategies  in  shotline  selections.  Providing  mobile  firepower  is  essentially  the  only  reason  an 
MBT  is  fielded.  Thus,  it  would  be  reasonable  in  both  strategies  to  devote  LFT  resources  to 
develop  a  solid  understanding  of  the  vulnerability  of  the  main  gun,  mobility  subsystems,  and 
supporting  subsystems. 

3.2.3  Damage  Assessment 

The  objectives  of  damage  assessment  following  ballistic  interactions  in  both  the  traditional 
platform-centric  and  MMF-based  SoS  task-focused  strategies  are  to  identify  (1)  platform 
component  damage  and  loss  of  subsystem  function,  (2)  crew/passenger  casualties,  and  (3) 
expedient  repairs  for  platform  restoration. 

In  the  traditional  platform-centric  strategy,  component  damage  and  subsystem  loss  (Level  2)  are 
mapped  to  combat  utility  (Level  4)  through  the  use  of  an  operations  research  mapping  tool,  such 
as  the  DAL.  In  the  MMF-based  SoS  task-focused  strategy,  however,  the  focus  is  the  capabilities 
of  the  platform  within  the  context  of  the  SoS  (Level  3)  and  the  ability  to  complete  SoS  mission 
tasks.  Mapping  component  damage  and  subsystem  loss  of  function  to  the  SoS  remaining 
capabilities  requires  analysis  and  operational-type  tests. ^  The  remaining  capabilities  following 
an  LFT  can  be  determined  in  laboratory  settings.  Quasi-operational  tests  using  companion 
vehicles  can  be  used  to  confirm  the  usefulness  of  residual  capabilities  within  the  SoS  construct  or 
to  develop  and  validate  “workarounds”  to  complete  mission  tasks  with  capabilities  remaining 
within  the  SoS  (i.e.,  damaged  platform  plus  companion  platforms). 

Consider  the  following  example.  With  the  traditional  platform-centric  strategy,  it  is  common 
practice  to  catalog  the  damage  to  platform  components  after  the  LFT  firing,  determine  their 
remaining  functionality,  and  assess  the  remaining  functionality  of  the  subsystems  containing  the 


It  is  recognized  that  in  the  traditional  platform-centric  strategy,  more  shots  may  be  devoted  to  certain  specific  capabilities 
because  our  knowledge  of  the  vulnerability  of  those  capabilities  is  less  than  for  others. 

^Crew  incapacity  would  be  mapped  first  to  platform  loss  of  capabilities  and  remaining  platform  capabilities.  The  results  of 
this  mapping  would  be  incorporated  into  the  laboratory  and  quasi-operational  tests  described. 
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damaged  components.  Determining  the  remaining  subsystem  functionality  has  generally  been 
done  in  a  laboratory  setting,  that  is  on  the  LFT  pad,  on  a  mobility  test  course,  or  at  another  fixed 
site  where  there  is  no  attempt  to  replicate  tactical  settings  or  operations.  In  a  laboratory  setting, 
vehicles  have  been  driven  on  a  test  track  to  determine  limits  of  mobility,  guns  have  been  fired, 
and  radio  transmission  or  reception  tests  have  been  made  using  other  vehicles  located  at  specific 
distances  from  the  tested  vehicle.  While  these  tests  are  useful  for  determining  degradations  in 
platform  capabilities,  they  seldom  give  evidence  of  the  significance  of  the  loss  of  platform 
capabilities  from  a  mission  perspective  or  indicate  whether  some  “workaround”  can  be  devised. 
Unit-level  or  force-level  models  are  typically  used  to  assess  significance,  but  seldom  is  there 
experimental  verification.  Further,  in  these  simulations,  the  engaged  system  is  considered  either 
fully  functional  or  killed  in  the  context  of  Level  4  combat  utility  metrics.  For  example,  a 
platform  with  damage  to  the  main  gun  is  considered  to  have  either  full  firepower  or  no  firepower 
at  all.  Seldom  is  a  partial  loss  of  system  capability  considered. 

In  an  MMF-based  SoS  task-focused  strategy  in  which  the  critical  issues  focus  on  the  platform’s 
contribution  to  SoS  mission  task  completion  (i.e.,  vs.  a  focus  on  the  capabilities  of  an 
autonomous  platform),  there  are  likely  to  be  instances  where  it  is  of  paramount  importance  to 
determine  the  significance  of  the  degradation  or  loss  of  certain  platform  capabilities  within 
specific  mission  settings.  Quasi-operational  tests  in  which  one  or  more  SoS  companion  vehicles 
operate  in  a  tactical  setting  with  the  damaged  vehicle  in  accordance  with  appropriate  doctrine, 
tactics,  techniques,  and  procedures  could  help  determine  whether  the  degradation  or  loss  of  a 
specific  capability  or  combination  of  capabilities  seriously  threatens  mission  success.  Such  a  test 
could  also  be  used  to  investigate  the  viability  of  “workarounds”  to  accommodate  the  weaknesses 
of  the  tested  platform.  For  example,  can  a  companion  vehicle  provide  total  or  partial  active 
protection  system  support  to  a  platfonn  that  can  neither  track  nor  fire  upon  incoming  munitions? 

Finally,  in  the  traditional  platform-centric  strategy,  an  important  part  of  the  damage  assessment 
process  is  identifying  the  expedient  repairs  that  can  be  made  to  restore  the  platform  to  some  level 
of  combat  utility.  The  BDAR  process  is  essentially  the  same  for  the  MMF-based  SoS  task- 
focused  strategy,  except  the  focus  is  on  restoring  platform  capabilities  that  are  needed  to 
complete  current  mission  tasks.  As  noted  in  section  3.2.1,  the  complexity  and  length  of  current 
combat  missions  in  the  SoS  environment  require  more  than  BDAR  to  support  long-tenn  repair 
considerations.  That  is,  mission  damage  assessment  and  repair  assessments  in  LFT&E  programs 
could  provide  insights  into  achieving  complex  repairs  of  components  and  subsystems  damaged 
in  combat  through  BDAR,  as  well  as  insights  into  accomplishing  incremental  repairs  as  the 
mission  progresses  through  the  use  of  other  maintenance  processes. 

One  possible  approach  is  the  development  of  methods  (by  either  the  crew  or  field  repair  units)  to 
identify  potential  repair  complexities  induced  by  the  ballistic  attack  and  the  parts  and  skill 
requirements  needed  to  effect  extensive  repairs  of  those  components.  For  instance,  although  it 
may  be  a  rather  simple  matter  to  replace  a  black  box,  ballistic  damage  to  the  sheet  metal,  the 
attachment  fittings,  etc.,  could  make  replacement  of  the  box  much  more  difficult.  Thus,  the  need 
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to  make  ancillary  repairs  of  extensive  ballistic  trauma  increases  the  overall  scope  of  the  repair 
and  requires  specific  parts  and  skills  to  effect  the  black  box  replacement. 

Insights  gathered  through  MMF-based  LFT  mission  damage  assessment  and  repairs  are  useful  in 
repairing  and  restoring  platfonn  capabilities  degraded  as  the  result  of  a  spectrum  of  stimuli, 
including  nonballistic  events  and  nonnal  wear  and  tear  over  time.  The  MMF  taxonomy  and  V/L 
models  under  development  provide  a  framework  for  considering  (1)  the  effects  of  degradation  of 
components  from  all  causes  (e.g.,  reliability  failure,  in  addition  to  ballistic  damage)  on  the 
platform  capability  to  complete  mission  tasks  and  (2)  the  ability  of  the  unit  to  repair  or  restore 
lost  capabilities  through  expedient  repair  as  provided  through  BDAR  or  through  the  field 
replacement  of  components  in  other  maintenance  actions. 


4.  Cost-Effective  LFT&E 


A  structured  process  for  building  cost-effective  vulnerability  assessment  strategies  and 
vulnerability  LFT&E  programs,  based  on  the  MMF,  is  discussed  in  this  section.  The  described 
process  assumes  (1)  an  LFT&E  program  (with  FUSL  LFT  in  the  absence  of  a  waiver)  is 
mandated  for  a  single  platform  that  is  part  of  an  SoS  and  (2)  operational  requirements  for  the 
platform  have  established  the  minimum  levels  of  capabilities  required  of  the  platfonn  (as  part  of 
an  SoS)  to  complete  tasks  linked  to  specific  missions  in  realistic  combat  scenarios  of  the  joint 
environment.  Two  very  important  considerations  in  establishing  a  cost-effective  LFT&E 
program  are  the  risks  and  costs  associated  with  the  assessment  process. 

4,1  Identifying  the  Risks  in  Vulnerability  Assessment 

Certainly,  an  important  objective  of  vulnerability  assessment  is  to  minimize  the  likelihood  that  a 
significant  vulnerability  (i.e.,  vulnerability  resulting  in  personnel  casualties,  catastrophic  loss  of 
the  system,  or  failure  to  complete  mission  tasks)  will  remain  undetected  in  a  fielded  platform, 
despite  the  actions  taken  during  the  acquisition  process  by  the  contractor,  the  Project  Manager 
Office  (PMO),*  the  T&E  Working-level  Integrated  Product  Team  (T&E  WIPT),-!  or  the 
independent  testers  and  evaluators.  The  risk  associated  with  this  likelihood,  a  vulnerability 
assessment  risk,  has  three  elements:  the  inherent  risk,  the  control  risk,  and  the  detection  risk 
(see  figure  9).! 


* 

The  PMO  has  the  “responsibility  for  and  authority  to  accomplish  program  objectives  for  development,  production,  and 
sustainment  to  meet  the  user’s  operational  needs  (DOD  Directive  5000.1,  p.  2  [2003]). 

WIPT  consists  of  headquarters  and  component  functional  personnel  who  support  the  materiel  developer  and  focus  on  a 
particular  such  as  T&E.  The  T&E  WIPT  is  a  subgroup  of  the  Integrating  Product  Team  and  produces  both  the  T&E  Strategy  and 
the  T&E  Master  Plan  (TEMP)  for  the  weapon  system  (Anny  Regulation  73-1,  p.  60  [2004]). 

+Risk  terminology  is  borrowed  from  a  financial  auditing  context,  although  use  of  tenns  is  not  strictly  analogous  to  the 
terminology  in  auditing  contexts. 
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Risk  and  Vulnerability  Assessment 


Minimize  risk  that  a  material/significant  vulnerability  will 
exist  undetected  in  a  fielded  system 


Inherent  Control  Detection 


Figure  9.  Three  elements  of  vulnerability  assessment  risk. 

The  three  elements  of  vulnerability  assessment  risk  can  be  described  as  follows:* 

Inherent  risk  is  the  susceptibility  of  the  weapon  system  to  material  or  significant  system  and 
personnel  vulnerabilities.  Assessing  the  level  of  inherent  risk  associated  with  a  system  requires 
an  understanding  of  the  environment  in  which  the  system  must  function,  the  expected  mission(s) 
of  the  system  and  the  tasks  required  to  ensure  mission  success,  the  threats  the  system  is  likely  to 
encounter,  and  the  potential  mitigating  effects  provided  by  other  platforms  of  the  SoS  and  the 
tactics  and  doctrine  of  the  system  users.  Information  relevant  to  the  vulnerabilities  of  similar 
models  or  prior  models  of  the  same  system,  the  complexity  of  the  design  of  the  system,  and  the 
impact  of  technological  developments  on  the  system  serve  as  input  to  the  assessment  of  inherent 
risk. 

Control  risk  is  the  risk  that  a  material  or  significant  vulnerability  will  not  be  prevented  or 
detected  in  analyses  and  tests  conducted  during  the  design  and  production  phases  of  the  system 
under  the  supervision  of  the  PMO.  An  accurate  assessment  of  the  level  of  control  risk  associated 
with  a  weapon  system  requires  an  analysis  of  the  elements  of  the  PMO-directed  survivability/ 
vulnerability  assessment  program  (e.g.,  design  and  engineering  analysis,  experimental  testing, 
inspection  during  production,  etc.)  and  the  critical  data  voids  addressed  by  this  program. 

Detection  risk  is  the  risk  that  a  material  vulnerability  will  not  be  discovered  by  analyses,  T&E,  or 
other  activities  conducted  prior  to  fielding  of  the  system  by  sources  independent  of  the 
manufacturer/contractor(s)  and  PMO.  The  level  of  detection  risk  is  managed  to  a  great  extent  by 
the  independent  system  testers  and  evaluators  who,  having  considered  the  levels  of  inherent  and 
control  risk  associated  with  a  system,  determine  the  nature,  timing,  and  extent  of  the  additional 
analyses,  tests  and  other  activities  (including  LFT&E)  to  be  performed  prior  to  full-rate 
production  and  fielding. 


* 


Adapted  from  material  presented  in  Nelson  (2000). 
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4.2  Weighing  the  Costs  of  Vulnerability  Assessment 

In  fielding  a  system  in  which  the  survivability  and  vulnerability  of  the  system  and  its  crew  are 
major  concerns,  control  costs  related  to  vulnerability  assessment  must  be  weighed  against 
failure-to-control  costs  (see  figure  10).* 


CONTROL  COSTS 


FAILURE-TO-CONTROL  COSTS 
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DETECTION 
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EXTERNAL 
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Figure  10.  Balancing  control  costs  against  failure-to-control  costs. 

Control  costs,  categorized  as  prevention  costs  and  appraisal/detection  costs,  are  those  costs 
incurred  in  the  design,  design  analysis,  and  T&E  phases  of  a  system  with  the  objective  of 
eliminating  or  reducing  the  presence  of  vulnerabilities  in  the  system.  Prevention  costs  are 
incurred  prior  to  and  during  the  production  process  to  plan  for  and  ensure  the  expected  level  of 
conformity  among  mission  needs,  operational  requirements,  design  specifications,  and  the  actual 
system  produced.! 

Appraisal/detection  costs  are  incurred  to  identify  any  nonconformity  of  the  system  in  production 
or  the  completed  system  to  the  design  specifications,  operational  requirements,  and  missions’ 
needs  prior  to  fielding  of  that  system.  Appraisal/detection  costs  include  the  costs  of  LFT&E  in 
which  the  system  is  tested  against  threats  likely  to  be  encountered  in  realistic  combat  scenarios.! 

Failure-to-control  costs,  specifically  internal  failure  costs  and  external  failure  costs,  are  costs 
associated  with  any  vulnerabilities  that  are  discovered  after  production  is  completed.  Internal 
failure  costs  include  the  costs  of  system  or  component  redesign,  engineering  changes,  and 
retrofit — costs  that  result  from  addressing  a  vulnerability  discovered  after  production  of  the 
system  but  before  the  system  is  fielded.  External  failure  costs  are  related  to  vulnerabilities 


In  a  manufacturing  context,  the  terms  “control  costs”  and  “failure-to-control  costs”  refer  to  costs  incurred  in  quality  control 
in  which  the  objective  is  the  minimization  of  defects  in  products  manufactured  and  the  satisfaction  of  users’  needs. 

^Methodology  assumes  that  operational  requirements  for  a  system  have  been  established  on  the  basis  of  the  capability  levels 
needed  for  mission  task  completion. 

fit  is  acknowledged  that  realistic  threats  employed  in  LFT&E  might  include  ovennatching  threats  or  threats  not  considered  at 
the  time  operational  requirements  were  established. 


24 


discovered  in  the  system  after  fielding  and  include  the  significant  losses  associated  with 
personnel  casualties,  weapon  system  damage  or  destruction,  and  the  failure  of  the  SoS  to 
complete  the  tasks  that  contribute  to  the  prosecution  of  the  mission  in  the  joint  environment.* 

In  general,  the  total  sum  of  prevention  costs,  appraisal  costs,  internal  failure  costs,  and  external 
failure  costs  decreases  with  increased  attention  to  control  activities.  An  increase  in  control 
activities  (i.e.,  design  analysis,  inspection  and  testing  prior  to  full  rate  production)  often  results  in 
an  increase  in  internal  failure  costs  (i.e.,  the  costs  to  eliminate  or  reduce  the  vulnerabilities 
discovered  in  control  activities)1'  and  a  much  larger  decrease  in  the  very  significant  costs 
associated  with  external  failures. 

The  very  nature  of  external  failures  and  the  costs  attached  to  them  motivate  decision-makers  to 
consider  and  invest  both  time  and  money  in  control  activities,  such  as  LFT&E.  In  the  current 
world  of  limited  resources  and  constrained  defense  budgets,  it  becomes  important  to  design  an 
LFT&E  strategy  that  insures  that  the  data  collected  are  relevant  to  the  needs  of  those  assessing 
system  vulnerability  and  that  the  collection  process  is  consistent  and  efficient,  considering  both 
time  and  costs  incurred. 

4.3  Identifying  Data  Required  for  Vulnerability  Assessment  in  Ballistic  Interactions 

To  conduct  a  cost-effective  vulnerability  LFT&E  program,  the  LFT&E  WIPT  begins  by 
identifying  the  data  required  by  system  evaluators  for  vulnerability  assessment  decisions 
(i.e.,  required  data  set)  and  the  subset  of  data  required  for  assessment  that  are  unavailable  or 
unable  to  be  relied  upon  (i.e.,  data  voids). 

To  identify  the  required  data  set  for  assessment  decisions,  a  clear  understanding  of  the  objectives 
of  vulnerability  assessment  is  needed,  as  well  as  the  assumptions  made  in  describing  the 
assessment  methodology.  The  vulnerability  assessment  methodology  defined  in  this  report 
assumes  that  mission  decomposition  into  lower-level  tasks  (possibly  having  a  necessary  start- 
completion  ordering)  has  been  completed,  a  relationship  between  the  lower-level  tasks  and  the 
minimum  levels  of  system  capabilities  needed  to  complete  those  tasks  has  been  established,  and 
the  platforms  and  SoS  that  provide  the  capabilities  to  complete  the  tasks  have  been  identified. 
Part  of  the  analysis  in  which  capabilities  are  linked  to  platforms  includes  an  identification  of  the 
redundancies  and  interdependencies  among  platforms  within  an  SoS.  The  methodology  also 
assumes  that  operational  requirements  for  the  platform  have  been  established  on  the  basis  of  the 
capabilities  required  for  identified  mission  tasks  that  are  part  of  realistic  combat  scenarios. 

Given  these  assumptions,  vulnerability  assessment  in  LFT&E  becomes  an  exercise  in  the 
evaluation  of  the  extent  to  which  the  interactions  of  the  identified  platfonn  and  the  ballistic 
threats  the  system  is  likely  to  encounter  in  combat  result  in  personnel  casualties  (i.e.,  personnel 


Costs  may  be  measured  in  nonmonetary  (e.g.,  crew  losses)  as  well  as  monetary  (e.g.,  hardware  destruction)  terms. 
1Failure  to  remedy  vulnerabilities  discovered  in  control  activities,  because  of  lack  of  time  or  resources,  may  also  result  in 
external  failure  costs. 
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vulnerability)  and/or  the  failure  of  the  system  to  retain  the  capabilities  that  were  determined  at 
the  time  of  acquisition  to  be  needed  to  complete  specific  mission  tasks  (i.e.,  system 
vulnerability).  The  operational  requirements  for  the  system  define  the  minimum  levels  of  those 
capabilities  that  must  be  retained  by  the  system  in  combat  for  task  completion  in  specific 
missions  and  provide  the  basis  for  detennining  the  data  required  for  the  vulnerability  assessment. 

Assessment  raises  the  question  (see  figure  1 1),  “Will  the  OWNFOR  platform  (i.e.,  system 
assessed)  deliver  the  minimum  levels  of  capabilities  identified  as  needed  as  part  of  an  SoS  and 
contribute  to  the  completion  of  SoS  collective  tasks,  given  the  interaction  with  likely  combat 
threats?”* 

The  minimum  levels  of  capabilities  identified  in  the  system’s  operational  requirements  are  linked 
to  specific  tasks  established  in  the  decomposition  of  multiple  missions.  Given  the  minimum 
levels  of  specific  capabilities  needed  to  complete  critical  mission  tasks  (e.g.,  capabilities  in  areas 
of  mobility,  firepower,  communications,  etc.),  analysts  concerned  with  the  assessment  of 
vulnerability  seek  to  identify  the  following  relative  to  the  OWNFOR  system: 

(a)  Systems  and  subsystems  that  must  be  operationally  functional  (at  partial  or  full  level)  to 
produce  the  minimum  levels  of  capabilities  needed  for  task(s)  completion  within  the  SoS. 

(b)  Critical  components  of  the  subsystems  identified  in  (a)./ 

(c)  Type  of  damage  (i.e.,  personnel  casualties,  catastrophic  loss  of  the  system,  damage  to 
critical  components  of  the  system)  to  the  platform  expected  from  the  interaction  of  the 
system  within  the  SoS  and  OPFOR  threats  likely  to  be  encountered  in  combat. 

(d)  Extent  to  which  the  system  and  its  personnel  are  vulnerable,  given  the  damage  states 
identified  in  (c)  and  the  critical  components  of  the  system  identified  in  (b). 

To  provide  the  data  described  in  the  four  preceding  categories,  the  following  activities  grounded 
in  the  MMF  are  suggested:^ 

1 .  Define  the  set  of  initial  configurations  of  the  threat(s)  likely  to  be  encountered  in  combat 
and  system  just  prior  to  ballistic  interaction.  This  set  of  configurations  selected  considers 
the  mission  objectives,  the  physical,  military,  and  civil  environments  of  combat,  location 
and  time,  and  the  users’  DOTMLPF  (Levels  5,  6,  and  7). 


% 

In  a  similar  manner,  lethality  assessment  evaluates  the  likelihood  that  the  OPFOR  platfonn(s)  is  able  to  supply  the 
capabilities  at  the  minimum  levels  needed  to  complete  the  opposition’s  tasks,  given  the  interaction  of  opposition  platform(s)  and 
OWNFOR  munitions  (i.e.,  system  assessed). 

^A  system  is  composed  of  subsystems,  and  each  subsystem  is  a  set  of  components.  Critical  components  are  those  components 
that  if  lost  will  result  in  a  degradation  of  one  or  more  subsystem  functions,  and  consequently,  a  reduction  in  system  capability 
(Roach,  1993). 

■''Adapted  from  material  presented  in  Nelson  (2000). 

^Similar  data  relevant  to  the  OPFOR  threat-target  systems  must  be  collected  for  lethality  assessment. 
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Figure  11.  Assessing  capabilities  against  mission/task  requirements.* 

2.  Develop  the  damage  operators  expected  in  ballistic  interactions  of  the  system  and  threat. 
Consideration  would  be  given  to  effects  of  ballistic  penetration,  including  effects  of 
fragment  penetration  and  behind  armor  debris;  air  blast,  ballistic  shock,  and  ricochet 
phenomena;  secondary  and  cascading  damage;  fire,  toxic  fumes,  and  fire  suppression 
system;  and  multiple  synergistic  damage  mechanisms.  The  damage  operators  establish  the 
link(s)  between  the  initial  conditions  of  the  threat  and  system  prior  to  interaction  (Level  1) 
and  the  damage  state  vectors  of  Level  2,  representing  the  system’s  physical  status 
following  threat-system  interaction. 

3.  Determine  the  specific  n-tuple  damage  state  vectors  (Level  2)  that  are  mapped  from  the 
specific  points  of  ballistic  threat-system  interaction  defined  in  Level  1.  The  n  elements  of 
the  damage  state  vector  describe  the  status  (i.e.,  kill,  no  kill)  of  the  n  number  of 
components  of  the  system  following  a  threat-system  interaction.  This  activity  also 
identifies  potential  for  personnel  casualties  and  catastrophic  loss  of  the  system.! 


* 

Figure  modified  from  Tanenbaum  and  Bray  (2005). 

^Personnel  casualties  may  also  affect  the  probability  of  mission  completion  (i.e.,  specific  number  of  fully  functioning  crew 
members  may  represent  a  critical  component  of  system). 
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4.  Apply  mapping  operators  or  degraded  state  operators  that  establish  the  link(s)  between  the 
n-tuple  damage  state  vectors  (Level  2)  that  represent  the  damage  states  of  the  system 
following  threat-system  interaction  and  the  m-tuple  capability  state  vectors  (Level  3)  that 
represent  the  capabilities  that  remain  following  the  interaction. 

5.  Compare  the  capabilities  remaining  following  ballistic  threat-system  interactions  (activity 
no.  4)  to  the  capabilities  needed  for  completion  of  mission  tasks.  Typically,  capabilities 
required  for  task  completion  will  be  reported  in  terms  of  a  Boolean  expression,  stating  the 
logical  conditions  that  must  be  met  for  mission  task  completion. v 

6.  Recognizing  that  system  capabilities  (Level  3)  aggregate  from  subsystems  and  components, 
compare  the  specific  components  that  are  expected  to  remain  in  operation  following  the 
threat-system  interaction  (activity  no.  3)  to  the  specific  critical  components  that  must 
remain  in  operation  to  deliver  the  identified  capability  levels  required  for  SoS  task 
completion.'!  After  considering  redundancies  and  interdependencies  among  components 
and  among  systems  in  the  SoS,  this  activity  identifies  component  and  subsystem 
vulnerabilities  that  may  be  critical  to  completion  of  mission  tasks. 

Data  that  are  unavailable  from  reliable  sources  to  complete  these  activities  define  the  data  voids, 
the  subset  of  the  required  data  set  that  is  addressed  in  the  LFT&E  program.  The  LFT&E  program 
is  designed  to  fill  the  data  voids  identified  by  comparing  the  specific  set  of  data  required  for 
assessment  (required  data  set)  to  the  subset  of  the  required  data  set  available  from  reliable 
sources.  A  cost-effective  LFT&E  program  stresses  the  achievement  of  assessment  objectives 
within  the  constraints  of  resources  available. 

4.4  Collecting  Data  Required  for  Vulnerability  Assessment 

Cost-effective  LFT&E  programs  are  designed  with  an  optimal  combination  of  component-level, 
subsystem-level,  and  system-level  test  data,  with  consideration  given  to  cost,  availability  of 
hardware,  and  production  schedule.  The  evaluation  of  the  LFT  results  and  data,  as  well  as 
results  confirming  data  from  supporting  program  activities  (e.g.,  M&S,  CDE)  must  be  presented 
in  a  format  useful  to  decision-makers  concerned  with  accomplishing  mission  tasks  in  the  joint 
operating  environment. 

4.4.1  Establishing  Data  Voids  in  Required  Data  Set 

The  construction  of  an  effective  LFT&E  program  begins  with  a  solid  understanding  of  the  links 
between  Level  3  capabilities  and  Level  4  tasks.  At  this  point,  the  decompositions  of  relevant 
SoS  missions  into  lower-level  tasks  will  have  been  completed,  and  the  relationship  between  the 


*  . 

This  assumes  that  links  have  been  established  between  the  specific  tasks  identified  in  the  decomposition  of  multiple  missions 
and  the  capabilities  required  to  complete  those  tasks.  It  is  expected  that  required  capabilities  will  vary  among  tasks,  many  tasks 
will  require  multiple  system  capabilities,  and  that  expressions  of  capabilities  required  to  complete  tasks  will  often  be  written  in 
tenns  of  compound  conditions  (i.e.,  written  with  tenns  of  “and”  or  “or”). 

^Degraded  Capability  State  models  that  map  component  level  state  changes  to  platform-level  capabilities  and  the  ability  to 
complete  identified  tasks  are  in  the  early  stages  of  development. 
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identified  tasks  and  the  minimum  levels  of  capabilities  to  be  supplied  by  the  platform  will  have 
been  established  (03,4  in  figure  12).  Therefore,  assessment  planning  starts  with  the  knowledge  of 
both  the  minimum  levels  of  capabilities  required  for  specific  task  completion  and  the  effects  on 
the  SoS  tasks  of  multiple  missions  if  capabilities  fall  below  specific  identified  minimum  levels. 
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platform  to 
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Figure  12.  Required  data  set  for  vulnerability  assessment. 

To  conduct  the  six  activities,  described  in  section  4.3,  that  complete  the  assessment  objectives, 
the  following  data  are  needed:  (1)  data  relevant  to  Level  1,  the  initial  configurations  of  system 
and  likely  opposition  threats;  (2)  data  linking  points  of  system-threat  ballistic  interaction  to 
damage  state  vectors  (Oi,2  in  figure  12);  and  (3)  data  linking  damage  state  and  capability  state 
vectors  (02,3  in  figure  12).  To  supply  the  required  data,  analysts  collect  the  following  types  of 
data  from  defense  databases,  including  data  from  other  military  services;  system  contractors;  and 
defense  analysis,  testing,  and  evaluation  agencies:* 

•  Combat  data  relevant  to  damage  mechanisms,  system  damage,  and  residual  capabilities  of 
system  as  associated  with  the  identified  threats. 


Some  data  sources  are  from  Army  Regulation  73-1. 
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•  Results  of  prior  tests  of  materials,  components,  and  subsystems  of  earlier  and  current 
models  of  the  system  or  earlier  or  contemporary  models  of  systems  with  similar 
technologies. 

•  Results  of  prior  system-level  tests  of  earlier  models  of  the  system  or  earlier  or 
contemporary  models  of  systems  with  similar  technologies. 

•  Advanced  technology  and  concept  technology  demonstrations. 

•  Force  development  tests/experimentations. 

•  Warfighting  experiments. 

•  Design  analyses  of  the  system  with  consideration  given  to  the  new  materials  and 
technologies  incorporated  into  the  system. 

•  Engineering  analyses  and  controlled  damage  experiments. 

•  Failure  modes,  effects,  and  criticality  analyses  (FMECA). 

•  Modeling  and  simulation  runs  that  incorporate  the  system  description,  threat 
characteristics,  and  damage  mechanisms  expected  in  threat-system  interactions. 

•  Results  of  developmental,  operational,  and  production  qualification  tests. 

If  vulnerability  assessment  is  begun  early  in  the  acquisition  cycle,  some  of  the  data  included  in 
the  required  data  set  may  be  available  from  the  preceding  listed  sources  early  in  the  design 
process  of  the  vulnerability  assessment  strategy.  Data  not  available  or  data  for  which  analysts 
hold  little  confidence  are  identified  as  data  voids  and  form  the  basis  for  the  design  of  the  LFT&E 
program. 

4.4.2  Establishing  Priorities  Among  Data  Voids 

In  the  design  of  the  LFT&E  program,  limited  resources  dictate  that  not  all  data  voids  for  all 
combat  scenarios  considered*  can  be  explored  and  relevant  data  gathered.  With  data  needs 
(i.e.,  required  data  set)  linked  to  mission  tasks  through  the  MMF,  decision-makers  are  able  to 
understand  more  readily  the  risks  of  addressing  and  not  addressing  individual  data  voids.  Data 
voids  not  addressed  result  in  a  level  of  uncertainty  or  risk  related  to  completion  of  specific  linked 
mission  task(s).  Consideration  would  be  given  to  redundancies  and  interdependencies  between 
systems  within  the  SoS  in  establishing  priorities. 

Decision-makers  need  to  detennine  what  level  of  uncertainty  is  acceptable  and  adopt  an 
appropriate  methodology  to  prioritize  data  voids  identified  from  the  required  data  set.  Several 
objective  prioritization  methods  are  available,  such  as  the  Analytic  Hierarchy  Process  (AHP)  or 


*  . 

It  is  assumed  that  mission-to-task  decomposition  has  been  accomplished  prior  to  the  design  of  the  LFT&E  program  on  the 
basis  of  probable  combat  scenarios.  Combat  scenarios  that  are  only  remotely  possible  but  are  associated  with  costly  losses  may 
also  be  part  of  the  analyses,  if  resources  are  available. 
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Quality  Function  Deployment  (QFD),*  but  subjective  prioritization  of  data  by  those  with 
experienced  domain-specific  judgment  is  most  often  the  approach  employed. 

4.4.3  Identifying  Alternative  Elements  of  LFT&E  Program 

A  cost-effective  strategy  of  vulnerability  assessment  begins  at  the  Concept  and  Technology 
Development  Phase  and  continues  through  the  System  Development  and  Demonstration  and 
Production  and  Deployment  Phases  of  the  acquisition  process.  Elements  of  the  LFT&E  program 
may  include  coupon,  component-level,  subsystem-level,  system-level,  and  FUSL  LFT,  as  well  as 
other  activities  (i.e.,  data  sources)  that  support  LFT,  such  as  developmental  and  operational 
testing,  M&S,  design  and  engineering  analyses,  CDE,  system  integration  laboratory  experiments, 
and  other  activities  that  address  the  identified  critical  data  voids  relevant  to  assessment  decisions. 

Planning  for  vulnerability  assessment  early  in  the  acquisition  process  may  increase  the  number 
of  options  available  to  address  the  critical  data  voids  and  allow  assessment  activities  to  be 
completed  at  a  time  when  design  or  engineering  changes  are  more  feasible  and/or  more 
economical.  For  example,  in  addressing  some  data  voids  in  an  LFT&E  program,  test  articles, 
such  as  mock-ups  or  replicas,  may  be  used  in  technical  tests  to  gain  insight  into  design  and 
engineering  issues,  and  these  test  articles  are  less  expensive  than  the  realistic  test  articles 
required  in  FUSL  LFT.  Other  critical  data  voids  (e.g.,  cascading  and  synergistic  damage 
mechanisms),  however,  are  unable  to  be  addressed  until  later  in  the  production  process  when  the 
required  hardware  for  testing  becomes  available. 

The  initial  focus  of  an  effective  LFT&E  program  is  the  selection  of  plan  elements  that  best 
address  the  identified  and  prioritized  data  voids.  A  vulnerability  LFT&E  program  addresses 
those  data  voids  concerned  with  the  specific  effects  (e.g.,  ballistic  penetration,  air  blast,  ballistic 
shock,  secondary  and  cascading  damage)  of  platfonn-threat  ballistic  interactions  on  the  identified 
platform’s  critical  components,  including  its  personnel.1- 

Figure  13  provides  an  illustration  of  the  building  blocks  (e.g.,  FUSL  LFT)  of  an  effective 
LFT&E  program.  Construction  of  the  program  requires  consideration  of  the  (1)  potential  data 
sources  or  program  elements  (e.g.,  component-level  LFT,  CDE),  (2)  relevant  ballistic  interaction 
effects  (e.g.,  fire,  blast)  that  must  be  explored  in  vulnerability  assessment,  and  (3)  prioritized  data 
voids  relevant  to  critical  components  (e.g.,  identified  as  X  and  Y  in  figure  13). 


*  .  . 

A  description  of  AHP  and  QFD  and  a  discussion  of  the  applicability  of  these  tools  to  group  decision-making  can  be  found  in 
references  (Hauser  and  Clausing,  1998,  pp.  68-73;  Nelson,  1997;  Saaty,  1994,  1995). 

1  Although  LFT&E  tends  to  focus  on  the  effects  of  damage  mechanisms  on  a  platform’s  critical  components,  unexpected 
perfonnance  results  (i.e.,  Level  3)  have  also  been  identified  as  the  outcome  of  an  LFT  event.  Unexpected  performance 
consequences  may  result  from  the  surfacing  of  an  unanticipated  damage  mechanism,  or,  more  typically,  from  an  insufficient 
knowledge  or  understanding  of  an  02,3  mapping  relationship. 


31 


In  choosing  between  alternative  elements  (i.e.,  elements  that  are  able  to  address  the  same  critical 
data  void)  to  include  in  an  LFT&E  program,  consideration  is  given  to  (1)  the  capacity  of  each 
element  to  address  the  critical/prioritized  data  voids,  (2)  the  time  constraints  of  the  assessment 
schedule,  (3)  the  production  schedule  and  availability  of  system  hardware,  and  (4)  the  costs 
attached  to  the  execution  of  each  element  and  the  group  of  elements.  Before  adopting  any  one 
strategy,  designers  of  the  LFT&E  program  must  consider  the  tradeoffs  among  the  performance 
(i.e.,  capacity  of  elements  to  address  data  voids)  of  elements  chosen,  cost,  and  schedule,  as  well 
as  the  risk — that  level  of  uncertainty  associated  with  each  element’s  capacity  to  provide  the 
critical  data  within  the  time  constraints  of  the  assessment  process  at  the  identified  cost — 
associated  with  the  combined  group  of  assessment  elements.* 

Identification  of  the  individual  elements  of  the  LFT&E  program  and  the  activities  that  are 
conducted  as  part  of  each  element  is  the  starting  point  for  both  estimating  the  cost  of  an  LFT&E 
program  prior  to  plan  implementation  and/or  computing  and  reporting  the  total  cost  of  the 
program  post-implementation. 


* 

The  cost  methodology  proposed  in  this  report  incorporates  many  of  the  characteristics  of  the  Cost  as  an  Independent 
Variable  (CAIV)  Methodology  as  explained  in  Nelson  (2000). 
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5.  Measurement  and  Reporting  of  Costs 


Cost  is  an  important  consideration  in  both  the  design  of  the  LFT&E  program  (i.e.,  elements 
selected  for  LFT&E  program)  and  the  design  of  the  individual  elements  of  the  LFT&E  program, 
such  as  FUSL  LFT.  A  typical  LFT&E  program  has  a  building  block  approach,  beginning  with 
coupon  and  component-level  LFTs,  proceeding  through  subsystem  and  system-level  testing,  and 
culminating  in  the  FUSL  LFT.  In  this  section,  the  costs  of  FUSL  LFT&E*  are  examined,  but  it  is 
proposed  that  this  methodology  is  equally  applicable  to  the  analysis  of  the  cost  of  other  elements 
of  the  LFT&E  program.  The  identification  and  measurement  of  costs  according  to  a  consistent 
methodology  is  a  requirement  for  valid  comparison  of  alternative  elements  of  an  LFT&E 
program. 

5,1  Implementation  of  Activity-Based  Costing 

The  first  step  in  computing  the  cost  of  FUSL  LFT&E,  an  element  of  an  LFT&E  program,  is  the 
identification  of  the  components  or  activities  of  FUSL  LFT&E  and  the  adoption  of  a 
methodology  for  the  measurement  of  the  costs  of  those  identified  activities.  Activity-based 
Costing  (ABC),  a  methodology  that  focuses  on  activities  as  the  basic  cost  objectives,  is 
suggested  as  an  appropriate  methodology  to  cost  FUSL  LFT&E. 

Employing  ABC  methodology,  the  cost  of  FUSL  LFT&E  is  determined  by  summing  the  direct 
material  costs  associated  with  FUSL  LFT  (i.e.,  costs  of  test  assets)  and  the  indirect  costs  of  the 
resources  consumed  in  the  cross-functional  FUSL  LFT&E  activities.'*' 

Implementation  of  ABC  in  costing  FUSL  LFT&E  requires  the  following:* 

•  Identification  of  all  direct  materials  used  or  consumed  (i.e.,  test  assets)  in  completion  of 
FUSL  LFT. 

Example:  Direct  materials  include  test  articles,  spare  parts,  munitions,  and  targets. § 


The  cost  of  FUSL  LFT&E  includes  the  cost  of  the  evaluation  that  follows  the  FUSL  LFT.  The  evaluation  is  based  on  the 
data  gathered  in  the  FUSL  LFT,  as  well  as  data  from  component-level  and  subsystem-level  LFTs,  analyses,  experiments,  M&S, 
and  other  relevant  data-gathering  activities. 

1  Direct  costs  typically  include  costs  of  materials  and  labor  that  can  be  traced  in  an  economically  feasible  manner  to  a  cost 
objective.  Indirect  costs  are  costs  that  cannot  be  traced  to  the  cost  objective  in  an  economically  feasible  manner  and  must  be 
assigned  through  an  intermediary  link  that  is  identified  by  the  cost  methodology  employed. 

‘‘'For  a  more  detailed  analysis  of  the  costs  of  U.S.  Army  FUSL  LFT&E  activities,  see  Nelson  (2000). 

^Threats  investigated  could  include  gun-fired  projectiles,  missiles,  rockets,  and  mines  requiring  a  variety  of  launching/firing 
capabilities  (DA  Pam  73-1). 
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•  Identification  of  all  activities,  as  well  as  sub-activities,  that  consume  resources  in  FUSL 
LFT&E.* 

Example:  Activities  in  FUSL  LFT&E  include  planning,  M&S,  execution  of  FUSL  LFT, 
documentation  of  test  results,  and  evaluation. 

The  M&S  activity  includes  the  sub-activities  of  model  extension/expansion  to  incorporate 
the  test  article,  munitions,  target(s)  of  interest,  and  damage  mechanisms  not  previously 
included  in  the  model,  as  well  as  improvements  in  damage  mechanisms  already  part  of 
the  model;  model  verification  and  validation  of  existing  models;  exercise  of  model  in 
preshot  predictions  and  reruns  of  preshot  predictions;  and  provision  of  full-view  V/L 
estimates  for  use  in  evaluation. 

•  Identification  of  the  type  and  quantity  of  each  resource  consumed  by  each  sub-activity  of 
each  identified  activity,  as  appropriate.!' 

Example:  Resources  used  in  the  test  execution  activity  of  the  FUSL  LFT  on  a  test  range 
include  labor,  equipment,  and  test  range  facility  use.  Data  gathered  for  each  sub-activity 
of  the  test  execution  activity  include  the  hours  worked  by  each  specific  level  of  skill 
required  to  complete  the  identified  sub-activity,  the  type  and  pattern  of  use  of  the 
instrumentation  employed, !■  and  the  materials  consumed  by  instrumentation  or  test  range, 
as  appropriate.  Data  relevant  to  materials  and  labor  used  in  the  protection  of  the 
environment  and  security  of  test  assets,  as  well  as  in  the  repair  of  test  assets,  would  also 
be  collected.5 

•  Computation  of  the  costs  of  all  resources  consumed  by  each  sub-activity  of  an  activity. 

Example:  Cost  of  labor  of  personnel  engaged  in  specific  sub-activities  of  the  test 
execution  activity  is  computed  by  summing  the  respective  products  of  estimated  hours 
projected  for  each  skill  and  average  wage  rate/hour  established  for  the  identified  skill. 

Computation  of  the  total  cost  of  a  sub-activity  requires  summation  of  the  costs  of  all 
resources  (e.g.,  labor,  materials,  equipment  usage)  consumed  in  that  sub-activity. 

•  Computation  of  the  total  cost  of  each  activity  that  consumes  resources  by  summing  the  cost 
of  all  sub-activities  identified  as  components  of  that  activity. 


* 

In  the  context  of  ABC,  the  term  "activity”  is  used  to  define  the  discrete  unit  of  work  for  which  costs  are  to  be  identified,  and 
components  of  the  activity  are  identified  as  sub-activities  or  tasks.  In  this  report,  the  term  "sub-activities”  is  used  to  avoid 
confusion  with  military  "tasks,”  the  fundamental  building  blocks  of  missions. 

fin  implementation  of  ABC,  not  all  sub-activities  are  identified  and  costed  in  detail,  as  explained  in  section  5.3. 

+Pattern  of  use  describes  how  the  equipment  is  used  (e.g.,  for  single  FUSL  LFT  or  multiple  FUSL  LFTs). 

5Some  materials  used  in  repair  of  test  assets  may  be  classified  and  reported  as  direct  materials,  but  other  material  repair  costs 
may  be  classified  and  reported  as  part  of  the  costs  of  the  activities  with  which  they  are  associated.  Classification  is  related  to  ease 
with  which  cost  can  be  traced  to  test  asset. 
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•  Computation  of  the  cost  of  FUSL  LFT&E  element  by  summing  the  identified  direct  costs 
and  the  total  costs  of  individual  activities  required  for  conducting  FUSL  LFT&E. 

The  advantage  of  employing  ABC  to  cost  the  elements  of  an  LFT&E  program,  as  well  as  the 
individual  activities  and  sub-activities  of  each  element,  is  that  ABC  provides  a  means  to  weigh 
the  value  added  against  the  costs  incurred  for  each  element,  activity,  and  sub-activity.  ABC 
affords  the  framework  to  identify  elements,  activities,  and  sub-activities  that  are  non-value-added 
and  are  able  to  be  eliminated  as  well  as  elements,  activities,  and  sub-activities  that  are  value- 
added  but  are  able  to  be  made  more  efficient.  In  other  words,  ABC  allows  a  person  to  look  at 
elements  of  an  LFT&E  program  both  from  a  strategic  viewpoint,  “Are  we  doing  the  right 
things?,”  and  from  an  operational  viewpoint,  “Are  we  doing  things  right?”  (Cooper  and  Kaplan, 
1991). 

Understanding  an  element’s  activities  and  sub-activity  components,  including  the  inputs  and 
expected  outputs  of  the  activities,  affords  decision-makers  an  opportunity  to  monitor  and  manage 
the  perfonnance  of  the  activities,  consider  the  resources  used  in  the  completion  of  the  activities, 
and  to  eliminate,  redesign,  and  improve  both  activities  and  sub-activities  contributing  to  those 
activities.  Determining  the  cost  of  conducting  an  element  of  an  LFT&E  program,  such  as  FUSL 
LFT&E,  appears  straightforward  —  implement  an  ABC  system,  compute  the  costs  of  the  sub¬ 
activities  of  the  activities  of  the  element,  and  sum  the  costs  of  test  assets  and  the  identified  sub¬ 
activities  across  activities.  There  are,  however,  some  complexities  in  ABC  implementation  that 
must  be  addressed  in  the  measurement  and  reporting  of  the  costs  of  the  test  assets  and  the  cost  of 
the  activities  of  an  LFT&E  program  element  such  as  FUSL  LFT&E. 

5.2  Complexities  in  Costing  Test  Assets 

FUSL  LFT  test  assets  include  the  following:  (1)  the  test  articles  (i.e.,  weapon  systems  tested), 

(2)  the  spare  parts  provided  for  the  test  articles,  (3)  the  munitions  fired  (i.e.,  applicable  to 
vulnerability  tests),  and  (4)  the  targets  fired  upon  (i.e.,  applicable  to  lethality  tests)  (Nelson, 
2000). 

5.2.1  Cost  of  Test  Articles 

The  costs  of  test  articles  have  been  accounted  for  in  various  ways  in  completed  LFT  programs, 
and  the  total  costs  of  FUSL  LFT&E  may  be  significantly  affected  by  the  choice  of  the  alternative 
reporting  value  chosen  for  the  cost  of  the  test  article(s). 

Alternative  reporting  values  for  test  articles  used  in  FUSL  LFT  include  (1)  the  cost  of  replacing 
the  test  article  with  an  article  identical  to  the  original  article  (i.e.,  replacement  cost),  (2)  the  cost 
of  returning  the  test  article  following  FUSL  LFT  to  its  condition  prior  to  FUSL  LFT  (i.e., 
restorative  cost),  or  (3)  the  cost  of  acquiring  the  original  test  article  (i.e.,  historical  cost). 

It  can  be  argued  that  the  restorative  cost  (i.e.,  the  cost  to  return  the  test  article  to  its  original 
condition)  is  the  most  appropriate  value  to  use  to  report  the  test  article  cost  if  (1)  the  test  article  is 
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in  a  salvageable  condition  following  testing,  (2)  the  restorative  cost  is  able  to  be  estimated,  and 
(3)  the  restorative  cost  is  the  smallest  of  the  three  proposed  alternative  values.  Regardless  of 
whether  or  not  the  test  asset  is  restored,  the  restorative  cost  is  a  cost  of  testing.  If  the  restorative 
cost  is  higher  than  one  or  both  of  the  remaining  alternative  values  or  the  test  article  cannot  be 
restored,  it  is  suggested  that  the  lower  of  the  alternative  values  be  reported  as  the  cost  of  the  test 
article.* 

All  available  alternative  values  for  costing  the  test  article  should  be  disclosed  and  made  part  of 
the  database  available  to  those  budgeting  for  the  FUSL  LFT  of  future  systems.  The 
computations  supporting  reported  values,  including  any  amounts  attached  to  these  costs  (e.g., 
engineering  support  costs),  should  be  explained  in  full  to  facilitate  the  processing  of  this 
information  in  comparative  analyses  of  assessment  plans  and  in  future  budgeting  activities. 

5.2.2  Cost  of  Spare  Parts 

The  cost  of  spare  parts  for  the  test  articles  needed  in  the  execution  of  the  FUSL  LFT  is  not 
always  easy  to  identify.  For  example,  the  contractor  may  agree  to  supply  spare  parts  as  needed 
for  a  fixed  cost  in  a  test  system  support  package  that  is  part  of  a  larger  contract,  such  as  an  LRIP 
contract.  The  attachment  of  the  support  package  to  the  LRIP  contract  may  make  it  difficult  to 
report  an  accurate  line  item  for  spare  parts.  In  other  cases,  spare  parts  may  be  supplied  by  the 
contractor,  and  a  contract  modification  following  the  completion  of  the  FUSL  LFT  accounts  for 
parts  required  of  the  contractor  during  the  testing  phase.  In  still  other  cases,  U.S.  Anny  testers 
acquire  parts  from  Army  depots. 

The  cost  incurred  at  the  time  of  purchase  (i.e.,  historical  cost)  is  generally  reported  as  the  cost  of 
spare  parts,  and  the  reporting  of  this  cost  should  include  a  description  of  the  contract  agreement. 
Disclosure  of  the  spare  parts  used  vs.  spare  parts  purchased  and  related  costs  may  prove  useful  in 
planning  of  subsequent  FUSL  LFT  events. 

5.2.3  Cost  of  Munitions  and  Targets 

Munitions  used  in  lethality  tests  or  targets  used  in  vulnerability  tests  are  often  acquired  in  prior 
combat  or  in  nonmonetary  trades  with  other  agencies.  It  may  be  difficult  to  obtain  an  historical 
cost  or  to  estimate  a  replacement  cost  for  the  munitions  used  in  lethality  tests  or  the  targets  used 
in  vulnerability  tests.  A  complete  accounting  of  the  costs  of  test  assets,  however,  would  include, 
if  available,  (1)  the  estimated  cost  of  the  munitions/targets  with  the  relevant  range  of  values  for 
that  estimated  cost  or  a  disclosure  of  the  inability  to  estimate  the  cost  for  identified  assets,  (2)  the 
cost  of  transportation  of  targets  to  test  ranges,  and  (3)  the  cost  to  repair  target  for  intended  use  or 
repair  following  test  use,  if  appropriate. 


* 

The  term  “restorative  cost”  is  used  to  describe  the  limited  situations  in  which  a  test  asset  is  able  to  be  restored  to  its  original 
condition.  If  a  test  article  is  restored  to  less  than  its  original  condition,  the  test  asset  will  likely  not  be  fielded  because  of  the 
potential  weakness  from  the  damage  incurred  in  testing.  The  test  asset  cost  may  then  be  reported  by  the  alternative  replacement 
or  historical  cost  less  the  net  salvage  value  of  the  restored  test  asset  (i.e.,  restored  value  less  cost  of  restoring). 
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5.3  Complexities  in  Costing  Activities 

There  are  several  questions  that  must  be  addressed  prior  to  costing  activities  in  the 
implementation  of  an  ABC  methodology.  First,  how  is  an  activity  defined?  Most  ABC  authors 
agree  that  an  activity  requires  a  verb  (i.e.,  action)  and  a  noun  (result)  that  characterizes  the 
activity  as  a  process,  such  as  conduct  preshot  predictions  or  participate  in  LFT&E  WIPT*  and 
must  represent  a  significant  level  of  expenditure  (Brimson  and  Antos,  1994).  Complexities  arise, 
however,  in  defining  the  activity  breadth  and  activity  depth  within  an  ABC  implementation,  such 
as  in  the  costing  of  the  activities  of  FUSL  LFT&E. 

In  the  implementation  of  ABC  for  the  FUSL  LFT&E  element,  activity  breadth  is  identified  by 
addressing  the  question,  “What  scope  of  activities  should  be  included  in  the  costing  of  FUSL 
LFT&E?”  Defining  the  activity  depth  in  ABC  implementation  provides  answers  to  questions 
such  as,  “How  much  detail  is  required  in  costing  task  components  of  identified  activities?” 
Delineating  activity  breadth  and  depth  in  ABC  implementation  requires  an  understanding  of  how 
decision-makers  plan  to  use  the  activity  cost  data  supplied  by  ABC  analysis. 

Assuming  the  purpose  of  ABC  implementation  is  the  production  of  reliable,  comparable 
(i.e.,  across  alternatives),  and  consistent  (i.e.,  across  periods)  cost  data  for  purposes  of  making 
strategic  and  operational  decisions,  it  is  helpful  to  identify  activity  breadth  by  categorizing  the 
LFT&E  program  element’s  activities  as  primary  or  secondary  relative  to  the  decision-makers’ 
proposed  use  of  the  ABC  data.  Primary  activities  would  be  defined  as  work  efforts  that  are 
directly  associated  with  the  cost  objective  of  decision-makers,  and  secondary  activities  as  those 
that  create  the  environment  that  allows  the  primary  activities  to  be  performed.  Primary  activities 
would  receive  priority  in  ABC  implementation  with  secondary  activities  included  if  decision¬ 
makers’  needs  so  dictated. 

In  reality,  conducting  a  U.S.  Army  ground  or  air  system  FUSL  LFT&E  program  results  in 
activities  in  many  agencies,  including  the  Office  of  the  Director,  Operational  Test  and  Evaluation 
(DOT&E),  the  Office  of  the  Deputy  Director,  Operational  Test  and  Evaluation,  Live-Fire  Testing 
and  Missile  Defense  (DDOT&E  [LFT]),  the  Office  of  the  Deputy  Under  Secretary  of  the  Anny 
for  Operations  Research  (DUSA  [OR]),  the  Office  of  the  Deputy  Chief  of  Staff  for  Intelligence 
(DCSINT),  the  U.S.  Army  Medical  Research  and  Materiel  Command  (MRMC),  the  U.S.  Army 
Training  and  Doctrine  Command  (TRADOC)  including  the  U.S.  Army  Ordnance  Center  and 
School  (USAOC&S)  and  the  U.S.  Army  Transportation  Center  and  School  (USATC&S),  the 
U.S.  Congress  and  its  subcommittees,  analysis  groups  utilized  by  the  aforementioned  offices 
(e.g.,  Institute  of  Defense  Analysis),  U.S.  Army  Developmental  Test  Command  (DTC), 

U.S.  Army  Research  Laboratory  Survivability/  Lethality  Analysis  Directorate  (ARL  SLAD), 
Aberdeen  Test  Center  (ATC)  and  other  test  centers,  U.S.  Army  Materiel  Systems  Analysis 


The  LFT&E  WIPT,  a  subgroup  of  T&E  WIPT,  is  formed  to  coordinate  planning  of  LFT&E  program.  The  group  is  chaired 
by  the  system  evaluator. 
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Activity  (AMSAA),  U.S.  Army  Evaluation  Center  (AEC),  and  the  PMO  associated  with  the 
specific  weapon  system  tested. 

To  make  a  valid  comparison  of  the  costs  of  LFT&E  elements,  decision-makers  must  be  provided 
with  cost  data  that  are  identified  and  measured  according  to  a  consistent  methodology  employed 
across  assessment  plans  and  elements  of  the  plan.  For  each  plan  element,  the  breadth  of 
activities  and  depth  of  sub-activities  included  in  implementation  of  ABC  must  be  identified. 
Table  2  shows  one  possible  combination  of  the  activities,  sub-activities,  and  resources  consumed 
in  activities  that  may  be  selected  for  ABC  implementation  for  the  costing  of  FUSL  LFT&E.  As 
shown  in  this  table,  the  breadth  of  implementation  is  defined  by  seven  primary  activities,  and  the 
depth  is  limited  to  the  separate  costing  of  sub-activities  for  three  of  the  seven  activities. 

In  addition,  for  each  resource  consumed,  the  methodology  used  in  computing  the  cost  must  be 
described.  Referring  again  to  table  3,  it  can  be  seen  that  labor  is  a  significant  resource  consumed 
in  all  FUSL  LFT&E  activities  across  all  agencies.  In  accounting  for  the  labor  costs  of  employees 
engaged  in  activities  in  an  agency,  it  is  important  to  understand  how  that  agency  measures  and 
reports  its  labor  costs.  For  example,  are  overhead  administrative  costs  of  the  employee’s  agency 
attached  to  direct  labor  hours?  Are  labor  costs  based  on  the  actual  hours  worked  or  the  hours 
estimated  to  be  needed  for  the  required  service? 

In  situations  in  which  there  are  alternative  ways  of  measuring  the  costs  of  an  activity,  the 
measurement  method  employed  should  be  identified  and  described.  Allocated  costs  should  be  so 
identified,  and  the  bases  of  allocation  or  the  rates  used  in  allocation  should  be  fully  explained.  In 
addition,  information  that  allows  the  decision-maker  to  compute  the  costs  in  an  alternative 
manner  should  be  disclosed.  This  allows  uniformity  to  be  established  in  reporting  costs  across 
systems,  agencies,  and  periods. 

5.4  Complexities  in  ABC  Implementation  for  LFT&E  Programs 

Costing  the  elements  of  LFT&E  programs,  under  an  ABC  methodology,  requires  a  firm  grasp  of 
the  activities  and  sub-activities  that  define  the  program’s  elements  and  reliable  cost  data  relative 
to  the  resources  employed  in  conducting  those  activities  and  sub-activities.  Comparing  the  costs 
of  LFT&E  program  elements  conducted  by  different  agencies  requires  the  establishment  of 
guidelines  or  standards  for  measurement  and  reporting  of  those  costs. 

Currently,  the  costs  of  the  elements  of  the  LFT&E  programs,  as  well  as  the  costs  of  the 
individual  activities  of  the  elements,  are  determined  in  conversations  between  the  PMO,  the 
LFT&E  WIPT,  and  the  agencies  responsible  for  conducting  the  program  element.  For  example, 
the  costs  of  conducting  M&S  activities  are  identified  in  a  dialogue  between  SLAD  and  PMO 
personnel.  Some  agencies  engaged  in  LFT&E  efforts  use  a  variant  of  ABC  to  measure  the  costs 
of  specific  services  perfonned  by  personnel  of  that  agency.  Although  single  agencies  may  apply 
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Table  2.  Activities,  sub-activities,  and  resources  consumed  in  FUSL  LFT&E. 


Activity 
to  Be  Costed 

Sub-Activities 
to  Be  Costed 

•k 

Resources  Consumed  in  Activity 

Meetings  of  LFT&E  Working 
Integrated  Product  Team 
(LFT&E  WIPT) 

— 

Labor  hours  of  team  members 

Modeling  and  Simulation 

Extension  or  expansion  of  existing 
models 

Verification  and  validation  of  models 
to  be  used  in  FUSL  LFT&E 

Exercise  of  models  in  pre-shot 
predictions;  reruns  as  needed 

Provision  of  full-view  V/L  estimates 
for  use  in  the  FUSL  LFT  evaluation 

Labor  hours  of  those  engaged  in 
extension,  verification,  and  validation  of 
models 

Labor  hours  of  those  engaged  in  exercise 
of  models  in  pre-shot  predictions 

Labor  hours  of  those  providing  full-view 
V/L  estimates 

Materials  used  in  M&S,  as  needed 

Development  of  LFT&E  Event 
Design  Plan  (EDP) 

— 

Labor  hours  of  lead  in  preparation 

Labor  hours  of  other  LFT&E  WIPT 
members 

Writing  of  Detailed  LFT&E 

Test  Plan  (DTP) 

— 

Labor  hours  of  tester/preparer  of  plan 

Labor  hours  of  reviewer  of  DTP,  when 
applicable 

Performance  of  FUSL  LFT 

Setup  and  execution  of  FUSL  LFT, 
including  instrumentation 

Operation,  maintenance,  and  repair  of 
test  assets;  target  repair  and 
maintenance 

Battlefield  damage  and  assessment 
and  repair 

Damage  assessment  and  casualty 
assessment 

Labor  hours  of  those  engaged  in: 
planning,  training,  setup,  execution  of 
FUSL  LFT 

Labor  hours  of  those  engaged  in 
preparation  of  test  range,  BDAR  and 
damage  and  casualty  assessment 

Materials  (not  test  assets)  &  equipment 
used  in  FUSL  LFT;  materials  used  in 
protection  of  environment 

Test  facilities  used 

Preparation  of  Documentation 

Preparation  of  damage  assessment 
shot  records 

Labor  hours  of  preparers  of  shot  records, 
detailed  damage  assessment  report 

Preparation  of  detailed  damage 
assessment  report 

Labor  hours  of  lead  in  DTR  and 

DTR  contributors/reviewer 

Preparation  of  detailed  test  report 
(DTR) 

Preparation  of  the  System 
Evaluation  Plan  (SEP)  and 
Independent  System  Evaluation 
Report  (SER)  (FUSL  LFT 
evaluation  is  a  component) 

— 

Labor  hours  of  preparers  of  SEP  and 

SER 

* 

Costs  of  facilities  and  equipment  (including  computers)  used  for  completion  of  LFT&E  and  non-LFT&E  activities  may  be 
allocated  to  LFT&E  activities  in  full  ABC  implementation. 


39 


consistent  accounting  principles  (e.g,  rules  for  cost  allocation)  across  platforms  and  periods  in 
costing  the  sub-activities  and  activities  perfonned  by  that  agency,  there  is  little  uniformity  in 
measuring  and  reporting  costs  across  agencies  and  Services  (Nelson,  2000).  This  lack  of 
uniformity  makes  it  difficult  to  compare  the  costs  of  alternative  elements  (i.e.,  elements 
completed  by  different  agencies)  considered  in  the  design  of  LFT&E  programs. 

In  addition,  there  is  no  common  database  that  currently  exists  to  collect  LFT&E  program  cost 
data  across  single  or  multiple  Services  in  a  format  that  would  be  useful  to  test  planners.  The 
establishment  of  such  a  database  would  require  the  identification  of: 

•  Specific  data  to  be  collected  and  reported  (e.g.,  include  only  cost  data  that  are  the 
responsibility  of  the  PMCfi;  include  cost  data  for  specific  identified  activities  and  sub¬ 
activities). 

•  Acceptable  cost  data  sources  to  be  used  for  costs  reported. 

•  Acceptable  methods  for  the  measurement  of  costs,  including  the  allocation  of  costs. 

•  Acceptable  format  for  reporting  data  (e.g.,  level  of  aggregation  of  costs). 

Contributors  to  the  database  would  be  required  to: 

•  Report  data  according  to  guidelines  provided  and  explain  incomplete  or  missing  data. 

•  Disclose  cost  measurement  and  allocation  methods  used,  including  bases  for  allocation,  if 
alternative  methods  are  acceptable. 

The  ability  to  share  activity  cost  data  across  systems  is  one  of  the  benefits  of  using  the  ABC 
methodology  to  cost  elements,  activities,  and  sub-activities  of  an  LFT&E  program.  For  example, 
the  cost  of  perfonning  a  sub-activity  that  is  common  to  the  test  execution  activity  of  multiple 
platforms  does  not  need  to  be  recomputed  for  each  system.  Minor  changes  in  the  sub-activity 
from  one  platfonn  to  another  (e.g.,  changes  in  wage  rates)  requires  only  minimal  modifications 
to  the  initial  cost  computations  made  for  the  first  system. 

Identifying  the  costs  of  activities  and  sub-activities  performed  in  the  completion  of  LFT&E 
programs,  the  proposed  data  base  would  provide  data  useful  for  the: 

•  Building  of  cost  estimation  models  for  budgeting  the  costs  of  future  LFT&E  program 
elements. 

•  Assessment  and  improvement  of  the  operational  efficiency  of  LFT&E  program  elements, 
activities,  and  sub-activities. 


^Thus,  the  costs  expended  by  AEC  in  evaluation  and  the  costs  incurred  by  USAOC&S  (other  than  travel,  which  is  reimbursed 
by  PMO)  in  BDAR  analysis  and  participation  in  damage  assessment  would  not  be  included. 
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•  Design  of  more  cost-effective  LFT&E  programs  by  weighing  the  value  added  against  the 
costs  incurred  in  completing  LFT&E  program  elements,  activities,  and  sub-activities.  The 
ABC  methodology  allows  the  identification  of  the  incremental  costs  incurred  with  the 
addition  of  specific  test  activities  designed  to  address  particular  data  void(s).  Incremental 
benefits  expected  from  additional  tests  would  include  a  more  complete  understanding  of 
platform  vulnerability  possibly  leading  to  product  modification  or  changes  in  military 
strategy. 


6.  The  Way  Ahead 


This  research  effort  builds  on  the  work  of  two  earlier  studies,  published  by  ARL,  in  which  the 
impacts  (i.e.,  costs,  benefits,  and  risks)  of  LFT&E  programs  are  addressed.  The  first  publication 
(Deitz  et  ah,  1996)  reported  the  results  of  the  SLAD/AMSAA  effort  to  develop  a  methodology 
that  would  quantify  the  advantages  and  disadvantages  of  conducting  FUSL  LFT  as  an  element  of 
an  LFT&E  program. t  Deitz  et  al.  (1996)  reviewed  the  V/L  process  and  V/L  assessment  models 
and  discussed  the  merits  and  limitations  of  four  different  risk-benefit  analysis  methods,  all  in 
early  stages  of  development. 

The  second  ARL  publication  (Nelson,  2000)  described  the  activities  of  the  FUSL  LFT&E  of 
three  different  weapon  systems,  identified  the  cost  components  of  those  activities,  and  proposed 
the  ABC  methodology  for  measuring  and  reporting  the  costs  of  LFT&E  programs.  Contributions 
of  FUSL  LFT&E  to  a  V/L  assessment  strategy  were  described,  and  suggestions  for  improving 
the  cost-effectiveness  of  FUSL  LFT&E  concluded  the  report. 

Although  acknowledging  the  contributions  of  prior  LFT&E  programs  in  the  areas  of  system 
design,  personnel  survivability,  and  development  and  validation  of  analytical  models,  this  effort 
recognizes  that  the  variety  of  threats  and  combat  landscapes  facing  today’s  warfighter  and  the 
increased  sophistication  of  platforms  operating  within  a  system  of  systems  context  in  an 
integrated  and  information-centric  battlefield  suggest  the  need  for  a  new  look  at  how  LFT&E 
programs  are  conducted.  Specifically,  it  identifies  changes  that  might  lead  to  a  more  efficient 
and  effective  LFT&E  process,  providing  decision-makers  with  more  relevant  and  reliable 
information  for  the  dollars  expended. 

An  MMF -based  SoS  task-focused  LFT&E  strategy  is  proposed  to  replace  the  traditional 
platform-centric  strategy  that  emphasizes  the  functional  capabilities  of  the  autonomous  platfonn. 
The  MMF-based  SoS  task-focused  strategy  focuses  on  the  extent  to  which  the  platform  retains 
those  capabilities  needed  for  completion  of  SoS  tasks  and  the  ability  of  the  SoS  to  complete 
current  and  future  mission  tasks  in  the  joint  environment. 


■^The  study,  commissioned  by  Mr.  Walter  Hollis,  the  Army  Deputy  Under  Secretary  for  Operations  Research,  was  tasked  to 
develop  a  methodology  to  improve  the  LFT  waiver  process. 
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The  MMF  provides  the  foundation  for  the  identification  of  the  critical  issues  and  the  design  of 
the  LFT  program  to  address  the  prioritized  data  voids,  as  well  as  the  design  and  execution  of  the 
evaluation.  Consideration  is  given  to  redundancies  and  interdependencies  between 
complementary  platforms  within  the  SoS,  and  the  importance  of  communications  across  systems 
is  recognized.  The  MMF-based  SoS  task-focused  strategy  provides  the  opportunity  to  use  test 
assets  (i.e.,  hardware,  range  time,  and  expertise)  to  address  V/L  issues  of  paramount  importance 
to  the  unit's  ability  to  complete  its  mission.  Because  the  MMF  provides  a  task-organizing 
process  that  links  mission  tasks,  capabilities,  and  available  resources,  decision-makers  are  able  to 
understand  more  readily  the  risks  in  specific  scenarios  of  addressing  and  not  addressing 
particular  data  voids. 

Critical  issues  emphasize  recoverability  in  the  MMF-based  SoS  task-focused  strategy,  with  the 
perspective  now  directed  toward  SoS  operations  in  the  joint  environment.  This  new  perspective 
would  be  expected  to  result  in  modifications  to  shotline  selection  and  damage  assessment,  as  the 
focus  expands  to  include  the  long-term,  as  well  as  short-term,  needs  for  SoS  capabilities.  It 
allows  a  realistic  assessment  of  technical  risk  associated  with  foregoing  test  shots  that  may  be  of 
interest  at  the  platfonn  level  but  are  not  critical  to  understanding  SoS  effectiveness.  In  a  roll-up 
of  platform  LFT  program  results  to  the  SoS  level,  decision-makers  are  provided  with  a  better 
grasp  of  the  ability  of  the  unit  of  operation  to  complete  tasks  to  standards  under  given  conditions 
and  the  risks  associated  with  alternative  courses  of  action.  LFT  results  are  analyzed  and 
evaluated  not  only  in  terms  of  the  platform  and  warfighter,  but  also  in  terms  of  network-enabled 
warfighting,  information,  and  interoperability. 

With  consideration  to  the  costs  and  risks  associated  with  vulnerability  assessment,  a  structured 
process  for  building  cost-effective  LFT&E  programs  in  an  MMF  enviromnent  is  presented.  This 
process  includes  the  identification  and  prioritization  of  data  voids  and  the  selection  of  the 
optimal  program  elements  for  addressing  those  voids,  considering  time,  production  schedule, 
hardware  availability,  and  cost.  ABC  is  proposed  as  the  appropriate  methodology  for  costing  the 
individual  elements  and  activities  of  an  LFT&E  program.  ABC  provides  a  framework  for  the 
decision-maker  to  view  LFT&E  program  elements  from  both  strategic  and  operational 
perspectives,  addressing  the  respective  questions,  “Are  we  doing  the  right  things?”  and  “Are  we 
doing  things  right?”  (Cooper  and  Kaplan,  1991).  It  affords  the  means  for  eliminating  non-value- 
added  elements  and  activities  and  seeking  ways  to  make  value-added  elements  and  activities 
more  efficient.  There  are,  of  course,  complexities  in  ABC  implementation,  and  suggestions  for 
addressing  those  complexities  are  discussed. 

Although  not  proposing  modifications  to  the  process  of  executing  LFTs,  the  described 
methodology  does  propose  significant  changes  to  the  process  of  planning  for  and  evaluating  the 
results  of  LFT&E  programs.  Implementation  of  a  cost-effective  MMF-based  SoS  task-focused 
approach  to  LFT  would  require: 
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•  The  integration  under  the  MMF  of  the  efforts  of  acquisition,  requirements,  M&S,  T&E,  and 
training  communities  within  and  across  Services,  achieved  only  through  the  support  of  top 
levels  of  defense  administration. 

•  The  allocation  of  resources  to  the  appropriate  Service  divisions  to  ensure  the  availability  of 
test  assets,  including  hardware,  testing  facilities/ranges,  and  people  with  the  levels  of 
expertise  needed  for  the  planning  and  evaluation  processes  of  the  proposed  LFT&E 
programs. 

•  The  construction  of  platform  operational  requirements  based  on  the  capabilities  needed  for 
the  completion  of  multiple  tasks  of  multiple  missions.  This  construction  requires  the 
definition  of  a  set  of  platform-appropriate  missions  with  links  established  between  mission 
tasks  and  the  levels  of  capabilities  required  to  complete  those  tasks. §  A  significant 
challenge  to  this  endeavor  will  be  the  definition  of  capabilities  needed  by  a  platform,  as  a 
component  of  an  SoS,  in  scenarios  in  which  some  or  all  SoS  components  are  operating  at 
less  than  their  full  capacities. 

•  The  identification  and  measurement  of  the  costs  of  LFT&E  elements,  according  to  a 
consistent  methodology  that  allows  the  value  added  in  completing  LFT&E  program 
elements  to  be  weighed  against  the  costs  incurred  in  conducting  those  elements. 

Although  implementation  issues  may  appear  daunting,  there  are  efficiencies  to  be  gained  by  the 
sharing  of  information  between  those  responsible  for  the  design  of  the  weapon  system,  the 
assessment  of  system  V/L,  the  training  of  system  users,  and  the  repair  and  maintenance  of  the 
system.  Understanding  the  mission  and  associated  SoS  tasks  is  the  basis  for  understanding  how 
users  rely  on  the  platform — the  foundation  for  building  cost-effective  acquisition,  T&E,  training, 
and  maintenance  programs.  Currently,  MMF  models  that  link  mission  tasks  to  capabilities  are  in 
the  early  stages  of  development.  These  models  are  expected  to  provide  a  foundation  for  the 
construction  of  operational  requirements  for  a  system,  as  well  as  for  the  assessment  of  the  V/L  of 
that  system  operating  within  an  SoS  environment. 

In  conclusion,  it  is  important  to  recognize  the  significance  of  the  LFT&E  program  to  the  total 
T&E  strategy.  The  objective  of  a  T&E  program  is  to  facilitate  the  measurement  and  assessment 
of  the  effectiveness,  suitability,  and  survivability  of  platforms  relevant  to  their  contributions  to 
the  SoS.  It  follows,  therefore,  that  an  LFT&E  program  needs  to  be  designed  and  conducted  to 
ensure  the  efficient  collection  of  reliable,  relevant  data  in  a  format  that  allows  evaluators  to 
assess  the  capabilities  of  the  platform  and  the  SoS,  of  which  the  platform  is  a  member,  to 
complete  identified  SoS  tasks  in  tactically  realistic  scenarios  following  ballistic  interactions.  The 
MMF  provides  a  valuable  structure  for  both  the  design  of  LFT  programs  and  the  evaluation  of 
LFT  results. 


sufficiently  large  and  representative  set  of  mission  scenarios  will  be  needed  to  satisfy  operational  requirements  and  T&E 
decision-makers,  as  well  as  the  user  community. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


ABC 

=  Activity-Based  Costing 

AEC 

=  U.S.  Army  Evaluation  Center 

AFTL 

=  Air  Force  Task  List 

AFV 

=  Armored  Fighting  Vehicle 

AHP 

=  Analytic  Hierarchy  Process 

AMSAA 

=  U.S.  Army  Materiel  Systems  Analysis  Activity 

APC 

=  Armored  Personnel  Carrier 

ARL 

=  U.S.  Army  Research  Laboratory 

ARL  SLAD 

=  U.S.  Army  Research  Laboratory  Survivability/Lethality  Analysis 
Directorate 

AoA 

=  Analysis  of  Alternatives 

ATC 

=  U.S.  Army  Aberdeen  Test  Center 

AUTL 

=  Army  Universal  Task  List 

BCT 

=  Brigade  Combat  Team 

BDAR 

=  Battle  Damage  Assessment  and  Repair 

CAB 

=  Combined  Arms  Battalion 

CAIV 

=  Cost  as  an  Independent  Variable 

CDE 

=  Controlled  Damage  Experiment 

DAL 

=  Damage  Assessment  List 

DCSINT 

=  Deputy  Chief  of  Staff  for  Intelligence 

DoD 

=  Department  of  Defense 

DOT&E 

=  Director,  Operational  Test  and  Evaluation 

DDOT&E  (LFT&E) 

=  Deputy  Director,  Operational  Test  and  Evaluation,  Live-Fire 
Testing  and  Missile  Defense 
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DOTMLPF 

=  Doctrine,  Organization,  Training,  Materiel,  Leader  Development, 
Personnel,  and  Facilities 

DTC 

=  U.S.  Army  Developmental  Test  Command 

DTP 

=  Detailed  Test  Plan 

DTR 

=  Detailed  Test  Report 

DUSA  (OR) 

=  Deputy  Under  Secretary  of  the  Army  for  Operations  Research 

EDP 

=  Event  Design  Plan 

FMECA 

=  Failure  Modes,  Effects,  and  Criticality  Analysis 

FCS 

=  Future  Combat  System 

FUSL 

=  Full-Up  System-Level 

FUSL  LFT 

=  Full-Up  System-Level  Live-Fire  Test 

FUSL  LFT&E 

=  Full-Up  System-Level  Live-Fire  Test  and  Evaluation 

IPT 

=  Integrated  Product  Team 

JCIDS 

=  Joint  Capability  Integration  and  Development  System 

JFC 

=  Joint  Forces  Command 

JLF 

=  Joint  Live-Fire 

LFT 

=  Live-Fire  Test  or  Live-Fire  Testing 

LFT&E 

=  Live-Fire  Test  and  Evaluation 

LFT&E  WIPT 

=  Live-Fire  Test  and  Evaluation  Working-level  Integrated  Product  Team 

LRIP 

=  Low-Rate  Initial  Production 

M&S 

=  Modeling  and  Simulation 

MBT 

=  Main  Battle  Tank 

MMF 

=  Missions  and  Means  Framework 

MRMC 

=  U.S.  Army  Medical  Research  and  Materiel  Command 

MTP 

=  Mission  Training  Plan 

OPFOR 

=  Opposing  Forces 

OPLAN 

=  Operation  Plan 
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OPORD 

=  Operation  Order 

OSD 

=  Office  of  the  Secretary  of  Defense 

OWNFOR 

=  Own  or  Friendly  Forces 

PMO 

=  Project  Manager  Office 

QFD 

=  Quality  Function  Deployment 

SEP 

=  System  Evaluation  Plan 

SER 

=  System  Evaluation  Report 

SoS 

=  System  of  Systems 

T&E 

=  Test  and  Evaluation 

T&E  WIPT 

=  Test  and  Evaluation  Working-level  Integrated  Product  Team 

TEMP 

=  T&E  Master  Plan 

TRADOC 

=  U.S.  Army  Training  and  Doctrine  Command 

UNTL 

=  Universal  Navy  Task  List 

UJTL 

=  Universal  Joint  Task  List 

USAOC&S 

=  U.S.  Army  Ordnance  Center  and  School 

USATC&S 

=  U.S.  Army  Transportation  Center  and  School 

V/L 

=  Vulnerability/Lethality 

WIPT 

=  Working-level  Integrated  Product  Team 

48 


NO.  OF 

COPIES  ORGANIZATION 


1  DEFENSE  TECHNICAL 
(PDF  INFORMATION  CTR 
ONLY)  DTICOCA 

8725  JOHN  J  KINGMAN  RD 
STE  0944 

FORT  BELVOIR  VA  22060-6218 

1  US  ARMY  RSRCH  DEV  & 
ENGRG  CMD 
SYSTEMS  OF  SYSTEMS 
INTEGRATION 
AMSRD  SS  T 
6000  6TH  ST  STE  100 
FORT  BELVOIR  VA  22060-5608 

1  INST  FOR  ADVNCD  TCHNLGY 
THE  UNIV  OF  TEXAS 
AT  AUSTIN 
3925  W  BRAKER  LN 
AUSTIN  TX  78759-5316 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
IMNE  ALC  IMS 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

3  DIRECTOR 

US  ARMY  RESEARCH  LAB 
AMSRD  ARL  Cl  OK  TL 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 


ABERDEEN  PROVING  GROUND 


1  DIR  USARL 

AMSRD  ARL  Cl  OK  TP  (BLDG  4600) 
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1  OUSD(AT)/S&T  AIR  WARFARE 

R  MUTZELBURG 
RM3E139 

3090  DEFENSE  PENTAGON 
WASHINGTON  DC  20301-3090 

1  OUSD(AT)/S&T  LAND  WARFARE 
A  VIILU 
RM3B1060 

3090  DEFENSE  PENTAGON 
WASHINGTON  DC  20310-3090 

1  UNDER  SECY  OF  THE  ARMY 
DUSA  OR 
W  HOLLIS 
RM  5D564 

102  ARMY  PENTAGON 
WASHINGTON  DC  20310-0102 

1  ASST  SECY  ARMY 

ACQSTN  LOGISTICS  &  TECH 
SAAL  ZP  RM  2E661 

103  ARMY  PENTAGON 
WASHINGTON  DC  20310-0103 

1  ASST  SECY  ARMY 

ACQSTN  LOGISTICS  &  TECH 
SAAL  ZS  RM  3E448 
103  ARMY  PENTAGON 
WASHINGTON  DC  20310-0103 

1  DIRECTOR  FORCE  DEV 
DAPR  FDZ 
RM  3A522 

460  ARMY  PENTAGON 
WASHINGTON  DC  20310-0460 

1  US  ARMY  TRADOC  ANL  CTR 
ATRC  W 
A  KEINTZ 

WSMRNM  88002-5502 

1  USARL 

AMSRD  ARL  SL  EA 
R FLORES 

WSMRNM  88002-5513 

1  USARL 

AMSRD  ARL  SL  El 
J  NOWAK 

FORT  MONMOUTH  NJ  07703-5601 


ABERDEEN  PROVING  GROUND 

1  US  ARMY  DEV  TEST  COM 
CSTE  DTC  TT  T 
APG  MD  21005-5055 

1  US  ARMY  EVALUATION  CTR 
CSTE  AEC  SVE 
R  LAUGHMAN 
4120  SUSQUEHANNA  AVE 
APG  MD  21005-3013 

12  DIR  USARL 

AMSRD  ARL  SL 
J  BEILFUSS 
P  TANENBAUM 
AMSRD  ARL  SL  B 
M  PERRY 

AMSRD  ARL  SL  BB 
S  JUARASCIO 
AMSRD  ARL  SL  BM 
D  BELY 

D  FARENWALD 
J  FRANZ 

AMSRD  ARL  SL  BD 
R  GROTE 

AMSRD  ARL  SL  BE 
L  ROACH 
AMSRD  ARL  SL  E 
M  STARKS 
AMSRD  ARL  SL  EC 
J  FEENEY 
E  PANUSKA 
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1  DPTY  UNDERSECRETARY  OF 
THE  ARMY 

S  T  BAGBY 
DASA  CE 

109  ARMY  PENTAGON 
RM  3E352 

2  DOT&E/OSD 
L  MILLER 

T  SHEPARD 

1700  DEFENSE  PENTAGON 
WASHINGTON  DC  20301-1700 

1  US  ARMY  TE  MGMT  AGENCY 
DACS  TE 
J  GARBER 

200  ARMY  PENTAGON 
RM2C139 

WASHINGTON  DC  20310  0200 

1  TECOM 
B  BARR 

PARK  CENTER  IV 
4501  FORT  AVE 
ALEXANDRIA  VA  22303 

1  DIR 

GEORGIA  INST  OF  TECH 
GTRI/ELSYSTEST  &  EVAL  RSCH  & 
EDU  CTR 

S  M  BLANKENSHIP 
ATLANTA  GA  30332  0840 

1  THE  MITRE  GROUP 

SEN  M&S  ENGINEER 
P  A  GLASOW 

1820  DOLLEY  MADISON  BLVD  W626 
MCLEAN  VA  22102 

1  JOINT  WARFARE  ANLYS  DEPT 
APPLIED  PHYSICS  LAB 
JOHNS  HOPKINS  UNIV 
W  M  KROSHL 
11100  JOHNS  HOPKINS  RD 
LAUREL  MD  20723  6099 

1  PROGRAM  AREA  MANAGER 
JOHNS  HOPKINS  UNIV 
APPLIED  LAB 
R  R  LUMAN 

11100  JOHNS  HOPKINS  RD 
LAUREL  MD  20723 


1  SCI  ADVISOR  TO  DIR 
OSD/DOT&E 
E  A  SEGLIE 

1700  DEFENSE  PENTAGON 
WASHINGTON  DC  20301-1700 

1  CHIEF  ENGINEER 

COMBINED  TEST  ORGANIZATION 
PM  FUTURE  COMBAT  SYS  BCT 
J  SHEEHAN 

1530  WILSON  BLVD  STE  1000 
ARLINGTON  VA  22209-2421 

1  LOGICON 
C  STANIEC 

2100  WASHINGTON  BLVD 
ARLINGTON  VA  22204-5706 

3  INST  FOR  DEFENSE  ANLYS 
B  TURNER 

L  TONNESSEN 

V  VOLPE 

1801  N  BEAUREGARD  ST 
ALEXANDRIA  VA  22311-1733 

4  SURVICE  ENGRG  CO 
D  VAN  DUSEN 

A  LAGRANGE 
E  EDWARDS 
R  VITALI 

4695  MILLENIUM  DR 
BELCAMP  MD  21017 

1  CHIEF  SCNTST  TECH  DIR 
AFOTEC  CN 
ML  WILLIAMS 

KIRTLAND  AFB  NM  87117-5558 

1  E  P  VISCO 

3752  CAPULET  TERRACE 
SILVER  SPRING  MD  20906 

1  HQ  SWC  AEA 
MAJ  BEERS 
730  IRWIN  AVE  STE  83 
SCHRIEVER  AFB  CO  80912-7383 

1  OFC  OF  THE  CIO/G6 

V  BETTENCOURT 

107  PENTAGON  RM  1A271 
WASHINGTON  DC  20310-0107 
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4  UNIV  OF  PITTSBURGH 

KATZ  SCHOOL  OF  BUSINESS 
J  BIRNBERG 
J  H  EVANS 
V  HOFFMAN 
D  MOSER 

PITTSBURGH  PA  15260 

20  FRANKLIN  AND  MARSHALL  COLG 
DEPT  OF  BUS  ORG  AND  SCTY 
M  K  NELSON 
BOX  3003 

LANCASTER  PA  17604-3003 

1  NAVAL  SUR  WARFARE  CTR 
CARDEROCK  DIV 
F  FISCH 

BETHESDA  MD  20084-5000 

1  FRANKLIN  AND  MARSHALL  COLG 

PROVOST 
B  PIPES 
BOX  3003 

LANCASTER  PA  17604-3003 

1  FRANKLIN  AND  MARSHALL  COLG 

DEPT  OF  BUS  ORG  AND  SCTY 
CHAIR 
A  GLAZER 

LANCASTER  PA  17604-3003 

1  DYNAMIC  RSRCH  CORP 
B  BRAY 

106  S  5TH  ST 

LEAVENWORTH  KA  66048 

2  DYNAMIC  RSRCH  CORP 
R  SMITS 

B  HARRIS 
60  FRONTAGE  RD 
ANDOVER  MA  01810 

1  THE  O  BRYON  GROUP 
J  F  O  BRYON 
1608  TOLLGATE  RD 
BEL  AIR  MD  21015-5825 

1  TECH  DIR  A 
J  MURPHY  JR 
46  OL/OGM/OL  AC 
2700  D  ST 

BLDG  1661  STE  C202 
WPAFB  OH  45433-7605 


1  TEXAS  ST  AT  UNIV  SAN  MARCOS 
DEPT  OF  COMPUTER  SCI 
F  HADDIX 
601  UNIVERSITY  DR 
SAN  MARCOS  TX  78666 

1  KENYON  COLLEGE 

DIR  OF  LAB  FOR  PHYS  AND  MATH 

J  T  KLOPCIC 

GAMBIER  OH  43022  9623 

1  JNT  AIRCRAFT  SURV  PRGM  OFC 
D  LINDELL 

1213  JEFFERSON  DAVIS  HWY 
CRYSTAL  GATEWAY  4  STE  1103 
ARLINGTON  VA  22202 

1  US  ARMY  RESEARCH  LAB 

AMSRD  ARL  SL  EC 
J  THOMPSON 
WSMRNM  88002-5513 

1  US  ARMY  RESEARCH  LAB 

AMSRD  ARL  SL  El 
D  LANDIN 

WSMRNM  88002-5513 

1  MITRE 

P  H  CHRISTENSEN 
2750  KILLARNEY  DR 
SUITE  100 

WOODBRIDGEVA  22192 

1  NORTHRUP  GRUMMAN 
TACTICAL  SYSTEMS  DIV 
J  ILLGEN 

130  ROBIN  HILL  RD  STE  200 
GOLETACA  93117 

1  US  ARMY  SPACE  &  MISSILE 

DEFENSE  COMMAND 
SMDC  RDTC  TED 
R  D  HOFFMAN 
POBOX  1500 

HUNTSVILLE  AL  35807-3801 

1  SURVICE  ENGRG  CO 
QUANTICO  CTR 
J  WALBERT 

3850  SETTLER  PARK  DR  STE  310 
DUMFRIES  VA  22026 
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1  ORSA  CORP 
W  YEAKEL 

1003  OLD  PHILADELPHIA  RD  STE  103 
ABERDEEN  MD  21001 

1  USATEC 

CSTE  AEC  CG 
J  STREILEIN 
4501  FORD  AVE 
ALEXANDRIA  VA  22302-1458 


ABERDEEN  PROVING  GROUND 


5  US AEC 

CSTE  AEC  SVE  BA 
R  WO  JCIECHO  W  SKI 
CSTE  AEC  SVE  BG 
P  SWOBODA 
P  PARIS 

CSTE  AEC  MOD  SIM 
R  MIRABELLE 
CSTE  AEC  SVE 
R  BOWEN 

1 1  DIR  USAMSAA 

P  DEITZ  (6  CPS) 

E  ATZINGER 
R  THOMPSON 
A  WONG 
D  HODGE 
B  PARIS 


S  POLYAK 
B  SMITH 

AMSRD  ARL  SL  BE 
G  DIETRICH 
E  FIORAVANTE 
C  KENNEDY 
M  MAHAFFEY 
K  MANNIX 
W  MERMAGEN 
A  PRICE 
J  SHINDELL 
R  ZIGLER 

AMSRD  ARL  SL  BM 
C  BARKER 
G  BRADLEY 
M  BURDESHAW 
P  FEDELE 
R  GANGLER 
R  SAUCIER 

AMSRD  ARL  SL  BS 
J  ANDERSON 
L  BUTLER 
E  DAVISSON 
D  HISLEY 
B  RUTH 
W  WINNER 


40  DIR  USARL 

AMSRD  ARL  SL 
R  SANDMEYER 
AMSRD  ARL  SL  EM 
D  BAYLOR 
AMSRD  ARL  SL  BB 
R  DIBELKA 
J  PLOSKONKA 
D  JORDAN 
G  KUCINSKI 
M  KUNKEL 
PKUSS 
G  MANNIX 
P  MERGLER 
J  SHORT 
B  WARD 

AMSRD  ARL  SL  BD 
A  DRYSDALE 
F  MARSH 
R  MOYERS 
L  MOSS 
J  POLESNE 
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